The idea is that if the user enters one or more characters that the Textbox will search its list of suggestions. In this test implementation this is done through a simple List<>. Ultimately my list will contain many thousands of items and I will need to replace the List<> with a more efficient search algorithm.

In the below example code typing a single “W” will expand to “Waterland”, if you continue typing beyond the final “d” it will suggest “Waterland Investments”. Type a “T” and it will branch to “Waterland Telecommunication Systems”.

using System;
using System.Collections.Generic;
using System.Text;
using System.Windows.Forms;

namespace Dijksterhuis.org
{
    class AutoSuggestControl : TextBox
    {
        List<string> Suggestions;
        int PreviousLength; 

        // V1.0 We are using a simple sorted list for the suggestions
        public AutoSuggestControl() : base()
        {
            Suggestions = new List<string>();

            // We keep track of the previous length of the string
            // If the user tries to delete characters we do not interfere
            PreviousLength = 0; 

            // Very basic list, too slow to be suitable for systems with many entries
            Suggestions.Add("Waterland");
            Suggestions.Add("Waterland Investments");
            Suggestions.Add("Waterland Telecommuncation Systems");
            Suggestions.Sort();
        }

        /// <summary>
        /// Search through the collection of suggestions for a match
        /// </summary>
        /// <param name="Input"></param>
        /// <returns></returns>

        private string FindSuggestion(string Input)
        {
            if (Input != "") 
            foreach (string Suggestion in Suggestions)
            {
                if (Suggestion.StartsWith(Input))
                    return Suggestion;
            }
            return null;
        }

        /// <summary>
        /// We only interfere after receiving the OnTextChanged event. 
        /// </summary>
        /// <param name="e"></param>
        protected override void OnTextChanged(EventArgs e)
        {
            base.OnTextChanged(e);
          
            // We don't do anything if the user is trying to shorten the sentence
            int CursorPosition = SelectionStart;
            if (Text.Length > PreviousLength && CursorPosition >= 0)
            {
                string Suggestion = FindSuggestion(Text.Substring(0, CursorPosition));
                if (Suggestion != null)
                {
                    // Set the contents of the textbox to the suggestion
                    Text = Suggestion;
                    // Setting text puts the cursor at the beginning of the textbox, so we need to reposition it
                    Select(CursorPosition, 0);
                }
            }
            PreviousLength = Text.Length;
        }

  

    }
}

This is a post from Martijn's C# Coding Blog.

Boxing in C# has little to do with Saturday night television but quite a bit more with that part-time job at the warehouse you had as a student. It is an important concept in C# that is related to how the compiler handles different kinds of variables in memory. Knowing how the compiler handles the various types allows you to avoid unexpected side effects in your code.

This article explains what boxing is, how it works and how it can negatively effect your code if you don’t pay attention to it. We also look at how generics can be used to improve your code’s efficiency. And we try to answer the ultimate question: Is everything in C# an object?

Everything is an object in C# .. but not all objects are created equally

In C#, everything inherits from System.Object, so they share all the common object methods :

string a = “hello world”;
int b = 34;
Console.WriteLine( “{0} {1}”,a.ToString(),b.ToString());

At first glance all objects look the same in C#. But an important distinction is made between value types and reference types. Value types are basic types such as struct, int, long, short etc. Reference types are all the classes you will use (strings, delegates, objects)

• Value types (int,struct..) etc are located on the stack. (unless they are part of a reference type, such as a class)
• Reference types (classes…) are accessed through a pointer to their actual location on the heap.

We simplify things a little for this article: The stack keeps track of what is executing in our application. As we enter and exit methods items are added to or removed from the top. The heap stores all the applications data, and since data can be added to and removed from it randomly it needs to be garbage collected.

Why the difference ? The stack is faster to access for the runtime so by putting simple types close at hand the code gains in efficiency.

Assigning variables

The graphic shows four possible scenarios :

1. Boxing: When you assign an integer to an object (object b=a): a new managed memory block is created on the heap. For this C# has to allocate memory at runtime.
2. By Value: If you copy the integer to another integer (int c=a) the value is simply copied to another memory slot on the stack (which is allocated at compile time), this is the fastest assignment as most checks can be done at compile time.
3. Unboxing: When you cast an object to a value type (d=(int)b) the result is stored on the stack. This is also an expensive operation as at runtime the value has to be retrieved from the heap and is then checked to see if the cast was valid. (If the source argument is null or a reference to an incompatible object, an InvalidCastException is thrown.)
4. By Reference: Classes are reference types so if you assign a new object to an older object (object e = b) no new instance is created, the object pointer on the stack simply points to the older object. A direct result of this is that if you modify either b or e, they will both reflect the change as they both point to the same location in memory.

Boxing is as simply putting a basic type in wrapper (making it a fully blown object), and unboxing taking that wrapped object and converts it back to a simpler type. To do the boxing managed memory needs to be allocated on the heap, references need to be updated, and the contents of the value type have to be copied.

Value types are copied, reference types just refer to the original object

The following code example shows how a simple change in code can have very different results.

    class MainClass
    {
        struct Demo
        {
            public int x;
            public Demo(int x)
            {
                this.x = x;
            }
        }
        
        public static void Main(string[] args)
        {
            Demo p = new Demo(10);
            object box = p;
            p.x = 20;
            Console.Write(((Demo)box).x);
        }
    }

   class MainClass
    {
        class Demo
        {
            public int x;
            public Demo(int x)
            {
                this.x = x;
            }
        }

        public static void Main(string[] args)
        {
            Demo p = new Demo(10);
            object reference = p;
            p.x = 20;
            Console.Write(((Demo)reference).x);
        }
    }

In Example A, because Demo is a struct (and thus a value type on the stack) when we box it a copy is made. When we modify the original no change is made to the copy.

In Example B, we have created Demo as a class. In this situation we can assign it to another object and we don’t need to box it. A reference is made instead. So when we update the original, the copy is also updated as they both point to the same location in memory.

Boxing and unboxing value types slows things down

Often you can’t rely on what the type of variable a function will take so you need to use an object variable as object is the lowest common denominator in .NET. In the following example we use the ArrayList class to store a set of integers. The ArrayList can store any type of variable, but to be able to do this it accept the object class.

        using System.Collections;

        public static void Main(string[] args)
        {
            int total = 0;
            ArrayList myList = new ArrayList();
            for (int Lp = 0; Lp < 10000000; Lp++)
                myList.Add(Lp); // Box: Integer to an object
            foreach(object item in myList)
                total += (int)item; // Unbox: Object to integer
        }

When we add an integer to the Arraylist it is boxed into an object, and when we retrieve it it is unboxed back into an integer.

Note that this is only an issue if we are trying to store value types in the arraylist. If we were trying to store objects (eg. classes) there is no boxing done, as the original type was an object already. In this situation a simple reference is stored. On retrieval there is no unboxing necessary as only a reference is returned.

Boxing in Action

When we look at our code in a dissassembler the boxing operation is clearly visible in the output stream:

; myList.Add(Lp); // Box: Integer to an object
IL_000f: ldloc.1
IL_0010: ldloc.2
IL_0011: box [mscorlib]System.Int32
IL_0016: callvirt instance int32 class [mscorlib]System.Collections.ArrayList::Add(object)

And when we convert the object back to an integer the reverse happens:

; total += (int)item; // Unbox: Object to integer
IL_0042: unbox [mscorlib]System.Int32

Generics remove the need to box and unbox value types

In .NET 2.0 generics and generic collections were introduced that remove the need to box and unbox variables in many common situations. The following example is functionally equivalent to the earlier boxing example. We use a generic here, but force it to be “object”:

        using System.Collections.Generic;
        public static void Main(string[] args)
        {
            int total = 0;
            List<object> myList = new List<object>();
            for (int Lp = 0; Lp < 10000000; Lp++)
                myList.Add((object)Lp); // We need a boxing operation
            foreach(int item in myList)
                total += (int)item; // Unbox back to an int
        }

The above example of course makes little sense because we are inefficient on purpose. I included it to demonstrate how with generics we can get around the need for boxing with generic collections.

        using System.Collections.Generic;
        public static void Main(string[] args)
        {
            int total = 0;
            List<int> myList = new List<int>();
            for (int Lp = 0; Lp < 10000000; Lp++)
                myList.Add(Lp); // No need to box things
            foreach(int item in myList)
                total += item; // No need to unbox
        }

Generic are similar to templates in C++, they allow you to specify a type and the compiler will generate all the required code. In our example we specify that our list is of type int, so the compiler will create a List class that supports integers. There is no longer a need to convert our integer type to an object first. If you would like to know more about how generics work have a look at another post I wrote on the subject: Generics in C# .

So how much of a difference does boxing make?

Boxing is slower but how much is the difference and do you need to care? To put things into perspective I tested the timing on the three code examples above. For this I used .NET’s Stopwatch class to measure the code’s performance. First of all, the .NET runtime is pretty fast and I found that I needed to set the loop iterations at some 10 milion before I was able to get a consistent result across runs.

• Boxing/Unboxing ListArray example: 2580 ms
• Boxing/Unboxing List<object> example: 2050ms
• Non boxing List<int> example: 825 ms

The non boxing List<int> example is about 2.5 times faster than List<object>, which is quite considerable.

To finish up…

Based on this should you rewrite all your code to implement generics? Probably not unless you have several tight loops that could use your attention, note that this only applies to value types (structs, ints) . For reference types (i.e. all classes) this is less of a problem as they are stored as a reference anyway.

It is however a good idea to implement generics in any code you write from here on. Not just because of the potential gains in speed but also because generics provide the compiler with much more information.

• Generics reduce the number of coding errors as you have compile-time checking of types
• Generics are more readable, you don’t need to cast all over the place and it’s always obvious what type it is associated with

Also if possible avoid passing value types as parameters to methods if these force a conversion to an object.

Image credit: Wall of boxed by celesteh

This is a post from Martijn's C# Coding Blog.

One of the best things about C# is that as the language and libraries expand thought is put into keeping things readable. Below I have listed 10 shorthands that you can use to make your code tighter and less wordy. No doubt you know one or more already — but do you currently use all ten of them ?

1. The ? conditional evaluation operator

I first read about the ? operator in the “The C Programming Language” book by Brian Kernighan and Dennis Ritchie (first published in 1978, but I didn’t get around to reading it until well into the second edition). So this isn’t anything new. But it IS handy. It allows you to compress a common if-then-else pattern into a single assignment.

int x = 10;
int y = 20;
int max;

if (x > y)
  max = x;
else
  max = y;

Using the (Question) ? Positive Answer : Negative Answer patterns the above can be rewritten as:

int x = 10;
int y = 20;
int max = (x > y) ? x : y;

2. Null-Coalesce operator (??)

How often do you test for null values in your code? Often? Then the null-coalesce operator (??) comes in handy. To see how it works consider the following code example:

object cache = null;
object d = new object();
object e;

if (c != null)
    e = c;
else
   e = d;

It is obvious that we can rewrite this using the single ? operator :

object cache = null;
object d = new object();
object e = (c != null) ? c : d;

Using the ?? operator we can make it even shorter. If the left hand side is null, the right hand side is assigned.

object cache = null;
object d = new object();
object e = c ?? d;

3. Object Initializers

After you create a new object you often have to assign one or more of its properties. With the introduction of C# 3.0 it is now possible to use object initializers to both improve the readability of this, and to shorten your code.

Customer c = new Customer();
c.Name = "James";

c.Address = "204 Lime Street";

can be written as:

Customer c = new Customer { Name="James", Address = "204 Lime Street" };

4. The using statement

Often you will need to allocate a system resource such as a font, file handle, network resource etc. Each time you need such a resource there are three critical steps to go through: You acquire the resource, you use it, and then you dispose of it. If you forget to properly dispose of it you will have created a memory or resource leak. This is best illustrated through the following patterns:

     // 1. Allocation of the object
     Font font1 = new Font("Arial", 10.0f);
     try
     {
      // 2. The bit where we use the resource
     }
     finally
     {
     // 3. Disposal of the object
     if (font1 != null)
     ((IDisposable)font1).Dispose();
     }

The using statement allows us to compress this down to:

// Allocate the resource
using (Font font1 = new Font("Arial", 10.0f))
{
    // The bit where we use the resource
}
// Disposal is automatic

The using statement is intended to be used with objects that implement the “IDisposable” interface which in practice is all .NET objects that allocate and manage resources.

5. Aliases for long winded namespaces and types

The names of C# identifiers can become quite long. If you are doing Microsoft Office automation in C# you might want to do something simple like open MS Word and change a document. You can use the “using” statement to create an alias for either a class or a namespace.

using Word = Microsoft.Office.Interop.Word;
...
Word.Application = new Word.Application() { Visible = True; }

6. Nullable objects

A variable needs to have a value, it cannot be null. Sometimes it would be handy it was possible to assign “null” (eg. undefined) to a variable. .NET 2.0 introduced the Nullable generic that makes this possible. The following two lines produce exactly the same object:

Nullable<int> x = null; 
int? x = null;

By putting a ? following a variable definition the compiler will wrap a Nullable<T> generic around the type.

7. Automatic Properties

C# 3.0 introduced automatic properties. A property typically consists of (but doesn’t have to) a private variable which is exposed to the outside world through getters and setters. The following is common example of this:

public class Person 
{
 private string _firstName;
 public string FirstName 
 {
    get { return _firstName; }
    set { _firstName = value; }
 }
}

From C# 3.0 on we can reduce the above to:

public class Person
{
 public string Firstname { get; set; }
}

The C# compiler will automatically generate a backing variable and the correct get and set properties. Why is this useful? After all you could have just made a string variable in the class public instead.

By defining it as a property allows you to add the actual validation logic to your class at a later stage. The in-memory signature of the class won’t change which means that any external libraries compiled against your code will not have to be recompiled.

8. Type inference

In typical C# you will always carefully spell out your definitions:

    string MyString = “Hello World”;

From the right side of the declaration it is obvious that only one type (string) will ever match this definition. So instead of us doing the work, why not let the compiler figure this out?

    var MyString = “Hello World”;

The above definition will also create a string variable named “MyString”. It is important to note that C# is still strongly typed.There is no performance impact results from using type inference.

The compiler does all the work figuring out the data type at compile time. Of course this feature created two opposite camps, one that thinks var should be liberally applied, and another one that abhors the whole idea. The middle ground seems to be that var should be used there were its use is obvious.

var SeniorStaff = Employees.Where(s => s.Age > 35);         
foreach(var Member in SeniorStaff)
           Console.WriteLine("Name: {0} Age: {1}",Member.Name,Member.Age);

For example what type would SeniorStaff be ?

((System.Linq.Enumerable.Iterator<<>f__AnonymousType0>)(SeniorStaff))

This should make you glad that the compiler figured it all out for you. I wrote more about this in another post: Anonymous Types : The Basics.

9. Lambda Expressions

C# 2.0 introduced anonymous methods, which are methods defined inside a method. Incredibly powerful and a nice way to put all kinds of evaluation logic inside your code they had the drawback that they could be quite hard to read.

Func mySeniorStaffFilter = delegate(int a) { return a > 35; };

The above method takes an integer as a parameter, and returns a boolean. It checks if the staff member passed to it is older than 35. If so, it returns true.

Lamba expressions make things a little easier to read, while being functionally exactly the same:

Func mySeniorStaffFilter = a => a > 35;

Even better, you can define them anywhere a delegate would have fitted:

var SeniorStaff = Employees.Where(s => s.Age > 35);

10. string.IsNullOrEmpty

Not really a language feature, but a useful little library function in its own right. How often do you need to test if a string is empty? Or null? The string.IsNullOrEmpty method returns true if this is the case.

if (String.IsNullOrEmpty(s) == true)
    return "is null or empty";
else
     return String.Format("(\"{0}\") is not null or empty", s);

Optional and named parameters in C# 4.0

As a bonus a little on the upcoming C# 4.0 which will introduce optional and named parameters. It is quite difficult to see what all the hoopla is about until you realize what optional parameters will do to improve Office automation coding in C#:

Before optional and named parameters:

Word.Application app = new Word.Application() { Visible = true; }
object missing = System.Reflection.Missing.Value;
object filename = @"c:\Document.docx";
object readOnlyValue = true;

app.Document.Open(ref filename, ref missing, ref readOnlyValue, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing, ref missing);

With the support of optional and named parameters this becomes:

Word.Application app = new Word.Application() { Visible = true; }
app.Documents.Open(@"c:\Document.docx",ReadOnly:true);

Microsoft Office certainly uses a lot of optional parameters. Note that in the first example all parameters are passed by reference. They still are in the second example. Behind the scenes the C# compiler creates the appropriate temporary variables to pass on, saving us from having to hand add them to the code.

To give proper credit, the above example comes from a video on this subject by Mike Ormond on Channel 9.

Image credit: Thinking About Something by Grazie Daverro

This is a post from Martijn's C# Coding Blog.

Ever watched an experienced programmer flick through code like there is no tomorrow? That is what caffeine addiction does to you. If you think that normal scrolling up and down isn’t fast enough you will find that a good development environment offers many keyboard shortcuts to let you work ever faster.

Keyboard shortcuts allow for things like indenting a selection, making it uppercase or lowercase, commenting it , jumping between bookmarks and moving lines of code up and down. These can be real time savers.

I was looking for a printable overview of available keyboard shortcuts for MonoDevelop but couldn’t find one so I have created my own.

If you are running a standard Linux distribution you will probably still be using the 1.0 version of MonoDevelop. If you compiled MonoDevelop 2.0 from source then you will be pleased to discover many new shortcuts. As the difference between 1.0 and 2.0 is significant I have made separate tables for the both of them.

• MonoDevelop 1.0/2.0 Keyboard Shortcuts (Open Office)
• Monodevelop 1.0/2.0 Keyboard Shortcuts (PDF)

This is a post from Martijn's C# Coding Blog.

Another topic he pushes is code readability and ensuring that all code statements are as clear as possible.

You will often have your own way of doing things. A shared coding guideline is a good way to ensure that each member of your team is writing code styled as similar as possible.

If your company is looking at adopting a coding guideline for its C# development this would be a very good place to start.

At 32 pages it might seem a little long at first but it reads quickly and most topics should already be familiar if you do a lot of coding. Its companion quick reference guide neatly puts most key items onto a single page and makes for a great handout.

This is a post from Martijn's C# Coding Blog.

1. Improving your code’s readability by documenting regular expressions
2. Creating conditional string replacement by using MatchEvaluators
3. Speeding up regular expressions by compiling them, caching them in memory and pre-compiling them to their own DLL.

If you are new to regular expressions in C# have a look at the theory of regular expression in Regular Expressions : The Basics. The second post Regular Expressions in C#: Practical Usage introduced the most common uses of regular expressions.

Documenting your Regular Expressions

Regular expressions can make for fine alphabet soup. The following expression validates an e-mail address and it does a good job at it. It is also very intimidating at first. So just imagine rereading your code after a few weeks, what is going on in there?

string validEmail = @"\b([a-zA-Z0-9._%+-]+)@([a-zA-Z0-9.-]+\.[a-zA-Z]{2,4})\b";

With a little squinting you see that I would like to extract two groups: the username part, and the domain name part. C# allows us to name each group to make things a little easier to read. We can use the ?<groupname> pattern to name each group.

A little rewrite can make our expression a lot easier to read. C# offers the “#” character to document our expressions in line.

       static string validEmail = @"\b    # Find a word boundary
                       (?<Username>       # Begin group: Username
                       [a-zA-Z0-9._%+-]+  #  Characters allowed in username, 1 or more
                       )                  # End group: Username
                       @                  # The e-mail '@' character
                       (?<Domainname>     # Begin group: Domain name
                       [a-zA-Z0-9.-]+     #  Domain name(s), we include a dot so that
                                          #  mail.dijksterhuis is also possible
                       .[a-zA-Z]{2,4}     #  The top level domain can only be 4 characters
                                          #  So .info works, .telephone doesn't. 
                       )                  # End group: Domain name
                       \b                 # Ending on a word boundary
                       ";

Because we have added a lot of spaces and new lines to our expression we need to tell Regex about them by specifying the RegexOptions.Multiline and RegexOptions.IgnorePatternWhitespace options.

          string testEmail = "martijn@dijksterhuis.org";
          Regex TestValidEmail = new Regex(validEmail,RegexOptions.Multiline | RegexOptions.IgnorePatternWhitespace);
           
           // Test the e-mail address
           Match TestResult = TestValidEmail.Match(testEmail);
            
           if (TestResult.Success)
           {
                Console.WriteLine("E-mail is: {0}@{1}",TestResult.Groups["Username"].Value,
                                                         TestResult.Groups["Domainname"].Value);
           }

Conditional string replacement

The RegEx.Replace method allows you to use substitution parameters to change the original content around. In a previous post we looked at how we could swap two words around by using grouped patterns and the $1 and $2 conditional replacement names.

Regex Replacer = new Regex(@"(\w*) (\w*)");
string Input = "Molly Mallone";
string Output = Replacer.Replace(Input,"$2 $1");
Console.WriteLine(Output);

That is sufficient if you just want to move the data around a little, but it would be nice if you could make a replacement conditional on some external condition. The Regex.Replace method allows you to specify a MatchEvaluator which does just that. MatchEvaluator is a delegate which takes Match as a parameter and returns the replacement string.

Handy for example if you are cleaning up a mailing list and want to conditionally update some, but not all, e-mail addresses. In the following code example we know that mail.dijksterhuis.org is now served by smtp.dijksterhuis.org, so we want to move all those users to the new domain name and leave all other e-mail addresses the same.

using System;
using System.Text.RegularExpressions;

namespace RegularExpression
{
	class MainClass
	{

    	static string validEmail = @"\b   			# Find a word boundary
							  (?<Username>			# Begin group: Username
							  [a-zA-Z0-9._%+-]+     #  Characters allowed in username, 1 or more
							  )                     # End group: Username
							  @					    # The e-mail '@' character
							  (?<Domainname>        # Begin group: Domain name
							  [a-zA-Z0-9.-]+        #  Domain name(s), we include a dot so that
                                                    #  mail.dijksterhuis is also possible
							  .[a-zA-Z]{2,4}        #  The top level domain can only be 4 characters
													#  So .info works, .telephone doesn't. 
							  )                     # End group: Domain name
                              \b
							  ";
		
		public static string UpdateDomainNames(Match match)
		{
			if (match.Groups["Domainname"].Value=="mail.dijksterhuis.org")
			 return match.Groups["Username"].Value + "@" + "smtp.dijksterhuis.org";
			return match.Groups[0].Value; // The original
		}
		
		public static void Main(string[] args)
		{

		   Regex TestValidEmail = new Regex(validEmail,RegexOptions.Multiline | RegexOptions.IgnorePatternWhitespace);
		   
		   string[] MailingList = new string[] { "martijn@dijksterhuis.org",
												 "user@mail.dijksterhuis.org",
												 "willy@wortel.org"};

		   foreach(string email in MailingList)
		   {
				// Conditionaly replace e-mail addresses 
				Console.WriteLine( TestValidEmail.Replace(email,UpdateDomainNames) );
		   }
		
		}
	}
}

Speeding up regular expressions by compiling them

Regular expressions can be quite slow and in another post I found that a simple string replacement routine was some 40 times faster than the equivalent regular expression. Often you will want to stick with the regular expression as it will save you many lines of coding.

As the RegEx class encounters your expressions it compiles them to an internal format. It steps through this internal format each time you query the expression. It is also possible compile your expression to MSIL (the byte code to which C# is compiled) directly. In the best possible scenario the Just-In-Time compiler then translates this MSIL code directly to machine code giving another speed boost to your expression.

A note of caution: According to the MSDN team the increase in speed can be up to 30% which is nice but certainly isn’t amazing.

You can do this by setting the RegexOptions.Compiled option when you create a new RegEx:

Regex theExpression = new Regex(thePattern,RegexOptions.Compiled);

The penalty for this is the time to compile the expression which can add significantly to your applications start-up time. So although “compiled” might sound faster it might actually be slower. This is best applied if you frequently use the expression and it has a very long lifetime.

The expression cache

If you use many regular expressions the RegEx cache is also an important factor in how quickly your code executes. Each time you define a regular expression the library needs to parse it. If you frequently use a small set of regular expressions they won’t be compiled over and over again, instead they come from a cache. You will find that .NET/C# caches the last 15 expressions. Any more and it will have to recompile them as it encounters them.

It is possible to expand the size of the cache by setting the Regex.CacheSize property to a higher value. This is probably best done after you made an overview of how many expressions are used by your code.

Compiling to an assembly

For compiling a regular expression to MSIL you need to pay a hefty price. But with your project about to ship it might be worthwhile to investigate taking your most frequently used regular expressions and putting them pre-compiled into a new assembly. The Regex.CompileToAssembly method performs this function. You will have to write a separate program to do the actual compilation, but once done you can link in the regular expression like any other assembly to your main application.

You can use the following class to create your own set of regular expressions and save them to a new assembly:

using System;
using System.Collections;
using System.Text.RegularExpressions;

namespace CompileExpression
{
	class MainClass
	{
		// Add the expressions to the hash table 
	 	public static Hashtable TheExpressions = new Hashtable();

		// CompileExpressions
		public static void CompileExpressions(string AssemblyName)
		{
			// Reserve space for each expression
			RegexCompilationInfo[] CI = new RegexCompilationInfo[TheExpressions.Count];

			int Cnt = 0;
        	foreach(DictionaryEntry de in TheExpressions)
        	{
				CI[Cnt++] = new RegexCompilationInfo((string)de.Value,		  // the reg. ex pattern
				                                     RegexOptions.Compiled,   // Options to specify
				                                     (string)de.Key,		  // name of the pattern
				                                     "TheRegularExpressions", // name space name
				                                     true );                  // Public? 
        	}

		   // Create a new assembly name structure
		   System.Reflection.AssemblyName aName = new System.Reflection.AssemblyName( );

		   // Assign the name
  		   aName.Name = AssemblyName;

		   // Compile all the regular expressions into the assembly
  		   Regex.CompileToAssembly(CI, aName);
		}
		
		public static void Main(string[] args)
		{
			// Add two expressions to the collection
			TheExpressions.Add("FindHTML",@"(<\/?[^>]+>)");
			TheExpressions.Add("FindTCPIP", @"(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})");

			// Compile them to my new assembly called "RegEx"
			CompileExpressions("RegEx");
		}
	}
}

This will create a file called “RegEx.dll” in the home directory of your program. The next step is to verify if this works as advertised. Create a new project in Visual Studio and add a reference (in the Solution Explorer right click the name of the new project and click “Add Reference…” and navigate to where the RegEx.DLL file is located.

The following class will load the FindTCPIP expression from the DLL and execute it:

using System;

namespace TCPSolution
{
   class Program
   {
       static void Main(string[] args)
       {
           TheRegularExpressions.FindTCPIP MatchTCP = new TheRegularExpressions.FindTCPIP();

           if (MatchTCP.Match("10.0.0.6").Success)
           {
               Console.WriteLine("This works!");
           }
       }
   }
} 

Regular Expressions and Mono

I tested, prodded and played with the code for these regular expression posts on MonoDevelop and Mono. With the exception of the final “Compile to DLL” example. The code for that example compiles but on execution it will throw an “Not Implemented” exception in Regex.CompileToAssembly.

The end

This ends the mini series of three posts on regular expressions. I hope you have enjoyed them. The previous posts in this series are:

• Regular Expressions : The Basics. The theory behind regular expressions.
• Regular Expressions in C#: Practical Usage Examples of common usage.

Image through Flickr by Djenan

This is a post from Martijn's C# Coding Blog.

This is the second post in the C# regular expression series and it follows up on “Regular Expressions in C# – The Basics” which explained the theory behind Regular expressions in C#. In this post we look at how to make practical use of regular expressions in our C# code.

This post touches on four major regular expression subjects:

• String Comparison – does a string contain a particular sub-string?
• Splitting a string into segments – we will take an IPv4 address and retrieve its dotted components
• Replacement – modifying an input string
• Stricter input validation – how to harden your expressions

String Comparison – finding valid HTML tags

One of the essential functions of expressions are their ability to find if a string is contained inside another one. The RegEx.Matches method tests if a given string matches the pattern.

We start with a simple example: finding out where the letter “a” is mentioned in a sentence:

            string Input = "apples make for great party accessories";
            Regex FindA = new Regex("a");

            foreach(Match Tag in FindA.Matches(Input))
            {
                Console.WriteLine("Found 'a' at {0}",Tag.Index);
            }

That was almost too easy. Regular expressions really shine if you don’t know exactly what you are looking for but you can describe it. In the following example we will look for all valid HTML tags in an input string.

What is a valid HTML tag? <code>, </code>, <b>,<img src=”">, </br> are all valid HTML tags.

Regex HTMLTag = new Regex(@”(<\/?[^>]+>)”);

To break this down:

1. All valid HTML tags start with a “<”
2. They might or not have a forward slash (we need to escape the forward slash) \/?
3. There is at least one or more characters which are not “>”
4. The tag ends with a “>”

The following code example searches for all valid HTML tags in the input string:

using System;
using System.Text.RegularExpressions;

namespace RegularExpression
{
    class MainClass
    {
        public static void Main(string[] args)
        {
            Regex HTMLTag = new Regex(@"(<\/?[^>]+>)");

            string Input = "<b><i><a href='http://apple.com'>Ipod News</a></b></i>";
            
            foreach(Match Tag in HTMLTag.Matches(Input))
            {
                Console.WriteLine("Found {0}",Tag.Value);
            }
        }
    }
}

Resulting in:

Splitting a string into parts

Parentheses () not only allow you to group your expressions into parts they allow you to split a single string into multiple segments which we can inspect individually. To demonstrate we will use a regular expression to split an IPv4 address into its components.

A decimal TCP/IP address looks like XXX.XXX.XXX.XXX with X being a decimal number. Each column has at least 1 digit, and a maximum of 3. So a single column can be described as “(\d{1-3})“. There are four columns, each seperated by a dot. The dot (.) has a special meaning in regex so we need to escape it. (\.)

The Regex.Match method returns a new Match instance. We can now test Match.Success to see if the input string matched the TCP/IP address pattern. Through the Match.Groups property can we then extract each of the four IP address columns.The zero entry in the Groups property is alway the complete match, in this case “10.0.0.6″. The [1] entry contains the first groups contents, [2] the second etc.

            string IPMatchExp = @"(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})";
            Match theMatch  = Regex.Match("10.0.0.6",IPMatchExp);
            if (theMatch.Success)
            {
                Console.WriteLine("{0}.{1}.{2}.{3}",theMatch.Groups[1].Value,
                                                      theMatch.Groups[2].Value,
                                                      theMatch.Groups[3].Value,
                                                      theMatch.Groups[4].Value);
            }

String Replacement

Often is useful to manipulate a string, by replacing the matched pattern with something new. The RegEx.Replace method allows us to specify a pattern to look for and a replacement string.

The following example matches the last character and space following each word and replaces it with “b_”.

            Regex Replacer = new Regex(@"\w "); // Single [a-zA-Z] followed by a space
            string Input  = "ax bx sax dam pom";
            string Output = Replacer.Replace(Input,"b_"); // Replace all items found with a b and underscore
            Console.WriteLine(Output);

Substitution Patterns

What to do if you would like to flip parts of a string? C# offers several substitution patterns for this. Substitution patterns can only be used in a replacement string, and are used in combination with grouping.

They are useful if you would like to format the results of the match. A common task is to flip two words around. In the below example we flip the name “Molly Malone” into “Malone Molly”:

            Regex Replacer = new Regex(@"(\w*) (\w*)");
            string Input  = "Molly Mallone";
            string Output = Replacer.Replace(Input,"$2 $1");
            Console.WriteLine(Output);

The regular expression is defined as two groups of words (\w*) separated by a space. Each group can be referred to with a substitution pattern. $1 refers to the first group, $2 to the second (and if we had defined more $3 would be the third etc).

Input validation – we have to be more strict

Often we need to check if the data inputed or read from a file matches a definition so that we know its valid. But for this to work we need to ensure that our expressions only match a valid input. Many expressions of convenience are defined too loose. If we are to use them for input validation we need to harden them.

The pattern we used in an earlier example neatly broke down a valid IP address. But it wasn’t very strict and there are many combinations that would have matched that aren’t valid IP addresses. 999.999.999.999 is not a valid IPv4 address but it would have matched our pattern (@”(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})”). So we couldn’t have used it for testing for a valid IP address.

So what is a valid match? We need to define this first.

A valid IP address range is from 0.0.0.0 to 255.255.255.255 (with each column being represented by a byte).

At this point there are two things we can do: we can validate the results returned by our expressions with a few additional lines of C# code or we modifying our regular expression to become stricter. As this post is about regular expressions we will modify our expression to match only valid IP addresses.

How do we define valid ? 0,9,10,19,100,199,200,249,255 are all valid inputs for each column. 300 isn’t valid, and neither is 299. To keep things simple, we don’t allow 09 as a valid input.

• Single digit: 0 – 9 :   [0-9]
• Double digit: 10 – 99: [1-9][0-9]
• Triple digit 1:  100 – 199:  1[0-9]{2}
• Triple digit 2: and 200 – 249:  2[0-4][0-9]
• Triple digit 3: 250 – 255 25[0-5]

The single ([0-9])and double digit ([1-9][0-9]) combinations can be combined into: [1-9]?[0-9]. (Read as: The first 1-9 is optional, occurs 0 or 1 time)

So a single column can be defined as: (([1-9]?[0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.) Note the “.” at the end.

On the final column we do not need a “dot”. We can save some space by repeating the first expression three times, but we need to write out the fourth in full. Thus our expressions becomes: ([1-9]?[0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.{3}([1-9]?[0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])

Not exactly easy to read, but lets test to see if it works as expected. The following example program tries all column combinations from 0-999

using System;
using System.Text.RegularExpressions;

namespace RegularExpression
{
    class MainClass
    {
        public static void Main(string[] args)
        {
            string IPTestExp = @"(([1-9]?[0-9]|1[0-9]{2}|2[0-4][0-9]|255[0-5])\.){3}([1-9]?[0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])";

            for (int Lp = 0; Lp < 999; Lp++)
            {
                string IPAddress = String.Format("{0}.{0}.{0}.{0}",Lp);
                
                if (Regex.Match(IPAddress,IPTestExp).Success)
                    Console.WriteLine("{0} is valid",IPAddress);
                else
                {
                    Console.WriteLine("{0} is invalid",IPAddress);
                    break;
                }
            }
          }
    }
}

For brevity the program ends at the first invalid combination. If we had let it run it would have shown 256-999 as invalid.

This took a bit of work but we now have a single line test to see if a string is a valid IPv4 address.

Concluding

This ends the second post in this series. In the next post I will look at some advanced regular expression topics.

If you would like to read more on the theory behind regular expressions have a look at the first post in the series: Regular Expressions in C# – The Basics

Image credit: Tambako

This is a post from Martijn's C# Coding Blog.

One of the most common coding tasks is to take an input, munch it around and turn it into something different altogether. Are you looking for FedEx numbers in a text file? Do you want to replace “love” with “hate” in your source files? Is a string a valid e-mail address? Problems like these can be solved by applying regular expressions, or “regex” for short.

Introduction

This post explores the basic theory of expressions. If you are already familiar with them and want to know how to use them in your own C# programs have a look at the next post “Regular Expressions in C# – Practical Applications”

Expressions offer a method of describing and testing for particular combinations of characters in a string. A simple regular expression can often save you from having to write many lines of regular code.

• Are you looking for the characters “car”  in “cartoon”, “carbonate” or  “carton” ?
• Do you want to only match when the word “car” is standing by itself as in  “car sales for 2009″ ?
• Or only return true when the car is red or blue ? “blue car”/ “red car” / “green car”

In C# expressions are provided by the RegEx class in the System.Text.RegularExpressions namespace.

The expressions themselves are more or less standard between computer languages. You can often take an expression from another language and with a little or no work apply them to your C# code. If you are not familiar with them yet you should consider learning to use them.

Regular Expressions to the rescue

What can you use regular expressions for?

• Data capture: split a string into multiple fields which you can manipulate. 13-Jan-2006 becomes (day,month,year)
• Data input validation: Check if the input followed the required formatting rules. For example test if a valid telephone number was entered.
• String comparison: Does A exist in B?
• String replacement: Replace “foo” with “bar”
• Code size reduction: One line of regular expression code can replace large amount of dedicated code

When not to use regular expressions?

Don’t use them when speed is of the essence. Expressions have a serious drawback in that they can be slow to execute. If you are concerned about optimizing a part of your code it can be worthwhile to write your own replacement. In a previous post I noticed that a simple string replacement routine was 40 times faster than the regular expression equivalent.

The basics

To understand expressions we need a little bit of theory. This bit explains all the main operators and how to use them.

Literal characters

The most basic expression contains a single character. If we define “c” as the expression and test it against “car company” it will match against the “c” in “car”. If we ask the RegEx class to search again it will match against the “c” in “company”.

Several characters have a special meaning: ?, +, *, \, [, ( , ), ], {, }, . (dot) and ^

If we want to include them we need to escape them first using a backslash:

• 10 * 10 = 100 wrong
• 10 \* 10 = 100 ok

Normally when parsing strings C# will try to break down escaped sequences such as \n,\r etc. Expression statements usually contain many backslash operators. By adding the “@” string literal the compiler will not inspect the string too much and take it literally instead.

Character Sets

Character sets allow us to limit the characters that can match. Say for example we want to use just the numbers 0-9: [0-9] , or the characters a-z & A-Z: [a-zA-Z]. A character set only matches against a single character, so the following doesn’t work: “c[a-z]kie” matches against “cokie” but not “cookie”.

You can also define your own sets. If you are matching a date, a date separator can be a defined as a space, dash or slash: [ -/]

Many character sets are used so often that they have been given their own shorthands:

• \w matches any word character [a-z,A-Z]
• \s matches any whitespace (space, tab)
• \d matches against any digit [0-9]

For a longer list of the available short hands have a look at my C# Regular Expression Cheat Sheet .

The Dot is special

The dot “.” matches against any character, except for line breaks. You should use it sparingly as it can introduce unwanted results. Often it is better to be more specific, using \w or \d, or a character set that limits the set of possible characters.

• “g..gle” matches “google”, “gaagle”,”g%$gle” and much more.
• “\d\d.\d\d.\d\d” matches a valid date such as “12-08-99″  and “12/08/99″ but also to an invalid date: “12508799″

Creating alternatives using the boolean “or”

A vertical bar separates (|) alternatives, so “red|blue car” would match either a red or blue car. Written in C# code:

if (Regex.Match(“blue car”,”blue|red car”).Success)
Console.WriteLine(“Matches!”);

You can add as many alternatives as you would like, so “red|blue|purple|yellow car” are all possible.

Grouping with parentheses ()

Parentheses () make it easier to group things  together. So if you would like to match for either “color” or “colour” you could write the word “color” (or “colour”) as one of:

• col(o|ou)r
• (color|colour)

Repetition

A repetition quantifier specifies how often a preceding element is allowed to repeat.

To give some examples:

• \d{1,3} reads as “a decimal digit (0-9)”, minimum of 1, maximum of 3
• [az]+ reads as “one or more of a-z”, “abc” matches, and so does “axxxz”

In the following example “aab” matches, but so does “aaab”.

Repetition is useful for testing if an input matches a required pattern. If you need to test for a telephone number formatted as : XXX-XXXX you could write this as \d{3}[-]\d{4}.

Lazy and Greedy matching

All the above repetition operators are “greedy”, they match to the longest possible string they can find.

• a[bz]+z against “abcbzcdze” returns “abcbzcdz“
• <a.+> against "<a href='index.php'>Beginning</a>" matches everything, instead of just the opening <a href"">.

To avoid this we can apply “lazy” matching instead. In a lazy match, as soon as it finds a match the parser stops and returns the result. You can make a match lazy by simply adding a question mark:

• a[bz]+?z against “abcbzcdze” returns “abcbz“
• <a.+?> against "<a href='index.php'>Beginning</a>" returns <a href='index.php'>.

Anchoring

All the above examples didn’t care where in the string the match was made. You could also use them repeatedly to find more instances of the match in the input string. Anchoring allows you to match only those strings that are close to the beginning and/or end.

• ^string reads as: only match if “string” is at the beginning of the input. The “^” indicates the beginning. So “string of wool” matches, but “woolly string” doesn’t.
• string$ reads as: only match if “string” is at the end of the input. Here the “$” indicates the end. In this case “string of wool” can’t match, but “woolly string” can.
• $string^ reads as: only match if “string” is the whole pattern. The “s” comes as the first character, and the “g” as the last. So only “string” can match this pattern.

This ends the theoretical introduction to Regular Expressions — see also the next post “Regular Expressions in C# – Practical Applications” .

Image credit: Sarae

This is a post from Martijn's C# Coding Blog.

If you would like to print this, click here for a pure HTML version.

Escape Characters

Alternation

Substitutions

Word boundaries

Quantifiers

This is a post from Martijn's C# Coding Blog.

One of my favorites in the PHP libraries is the strip_tags function. Not only does it neatly remove HTML from an input it also allows you to specify which tags should stay. This is great if you are allowing your visitors to apply some basic HTML tags to their comments. This post explores two issues: using C# to remove unwanted tags, and cleaning up unwanted attributes that might be hidden in the allowed tags.

I wanted to clean some comments posted to a website from unwanted HTML tags. The users are allowed to <B> or <I> and even <A href=”"></a> their posts but anything else must be stripped before it is posted to the site. I found several regular expressions for C# that allow you to strip HTML but these magically wipe all the HTML and leave nothing.

Below is the end result of of some hacking, and of course much love-hate with the regular expression library.

string StripTags(string Input, string[] AllowedTags)

The StripTags method takes an input string, and an array of allowed tags. It returns the input as a string, minus all not wanted tags.

string test1 = StripTags("<p>George</p><b>W</b><i>Bush</i>", new string[]{"i","b"});
string test2 = StripTags("<p>George <img src='someimage.png' onmouseover='someFunction()'>W <i>Bush</i></p>", new string[]{"p"});
string test3 = StripTags("<a href='http://www.dijksterhuis.org'>Martijn <b>Dijksterhuis</b></a>", new string[]{"a"});

Using the above example code returns the following:

string StripTagsAndAttributes(string Input, string[] AllowedTags)

The above StripTags function is similar to the original PHP strip_tags function in having the same weakness: It is still possible for a malicious user to insert attributes into each of the tags. Think “style=” and “id=”. We would be somewhat saver if we cleaned these as well. The StripTagsAndAttributes method does just that.

It first runs the input through StripTags, and for the remaining tags is strips out all but a restricted set of attributes.

string test4 = "<a class=\"classof69\" onClick='crosssite.boom()' href='http://www.dijksterhuis.org'>Martijn Dijksterhuis</a>";
Console.WriteLine(StripTagsAndAttributes(test4, new string[]{"a"}));

That “OnClick” attribute looks mighty unsafe. Running the above string through StripTagsAndAttributes as in the example above returns:

This function probably needs some tuning if you want to allow, or restrict things even further.

A word of caution

Regular expressions are voodoo, very cool, but still voodoo. The above functions work for the tests I have applied to them, but your mileage may vary! If you have a special situation that doesn’t work leave a note below and maybe we can work out the problems.

Credits

The strip_tags function is of course inspired by the PHP version , and a Javascript implementation thereof by Kevin van Sonderveld. The attribute stripping routine is based on the regular expressions by mdw252 in one of the strip_tags manual page comments.

Source code

The complete source code for the StripTags function and StripTagsAndAttributes function with my test code can be found below:

using System;
using System.Text.RegularExpressions;

namespace StripHTML
{
	class MainClass
	{
		
        private static string ReplaceFirst(string haystack, string needle, string replacement)
        {
       		int pos = haystack.IndexOf(needle);
            if (pos < 0) return haystack;
            return haystack.Substring(0,pos) + replacement + haystack.Substring(pos+needle.Length);
        }

		private static string ReplaceAll(string haystack, string needle, string replacement)
        {
             int pos;
			 // Avoid a possible infinite loop
             if (needle == replacement) return haystack;
              while((pos = haystack.IndexOf(needle))>0)
                       haystack = haystack.Substring(0,pos) + replacement + haystack.Substring(pos+needle.Length);
                        return haystack;
        }		

		public static string StripTags(string Input, string[] AllowedTags)
		{
			Regex StripHTMLExp = new Regex(@"(<\/?[^>]+>)");
		    string Output = Input;

			foreach(Match Tag in StripHTMLExp.Matches(Input))
			{
				string HTMLTag = Tag.Value.ToLower();
				bool IsAllowed = false;
				
				foreach(string AllowedTag in AllowedTags)
				{
					int offset = -1;

					// Determine if it is an allowed tag 
					// "<tag>" , "<tag " and "</tag" 
					if (offset!=0) offset = HTMLTag.IndexOf('<'+AllowedTag+'>');
					if (offset!=0) offset = HTMLTag.IndexOf('<'+AllowedTag+' ');
					if (offset!=0) offset = HTMLTag.IndexOf("</"+AllowedTag);

					// If it matched any of the above the tag is allowed
					if (offset==0)
					{
					 	IsAllowed = true;
						break;
					}
				}

				// Remove tags that are not allowed
				if (!IsAllowed) Output = ReplaceFirst(Output,Tag.Value,"");
			}

			return Output;
		}

		public static string StripTagsAndAttributes(string Input, string[] AllowedTags)
		{
			/* Remove all unwanted tags first */
			string Output = StripTags(Input,AllowedTags);

			/* Lambda functions */
			MatchEvaluator HrefMatch = m => m.Groups[1].Value + "href..;,;.." + m.Groups[2].Value;
			MatchEvaluator ClassMatch = m => m.Groups[1].Value + "class..;,;.." + m.Groups[2].Value;
			MatchEvaluator UnsafeMatch = m => m.Groups[1].Value + m.Groups[4].Value;
			
			/* Allow the "href" attribute */
			Output = new Regex("(<a.*)href=(.*>)").Replace(Output,HrefMatch);

			/* Allow the "class" attribute */
			Output = new Regex("(<a.*)class=(.*>)").Replace(Output,ClassMatch);

			/* Remove unsafe attributes in any of the remaining tags */
			Output = new Regex(@"(<.*) .*=(\'|\""|\w)[\w|.|(|)]*(\'|\""|\w)(.*>)").Replace(Output,UnsafeMatch);

			/* Return the allowed tags to their proper form */
			Output = ReplaceAll(Output,"..;,;..", "=");
			
			return Output;
		}
			

		public static void Main(string[] args)
		{
			string test1 = StripTags("<p>George</p><b>W</b><i>Bush</i>", new string[]{"i","b"});
			string test2 = StripTags("<p>George <img src='someimage.png' onmouseover='someFunction()'>W <i>Bush</i></p>", new string[]{"p"});
			string test3 = StripTags("<a href='http://www.dijksterhuis.org'>Martijn <b>Dijksterhuis</b></a>", new string[]{"a"});
			
			Console.WriteLine(test1);
			Console.WriteLine(test2);
			Console.WriteLine(test3);

			string test4 = "<a class=\"classof69\" onClick='crosssite.boom()' href='http://www.dijksterhuis.org'>Martijn Dijksterhuis</a>"; 
			Console.WriteLine(StripTagsAndAttributes(test4, new string[]{"a"}));
		}
	}

Image credit: Jesper Rønn-Jensen’s

This is a post from Martijn's C# Coding Blog.