www.lenholgate.com - Rambling Comments - Len Holgate's C++ progamming blog

Netmap, RIO and the challenges in using a 10 Gigabit pipe - Rambling Comments

2012-03-26T20:46:06Z

This link, Revisiting Network I/O APIs: The Netmap Framework, via highscalability.com makes for interesting reading. Especially given my current interest in the performance of the Winsock Registered I/O networking extensions, RIO , and the fact that my new network cards are so difficult to fully utilise.

Windows 8 Registered I/O Performance - 10 Gigabit networking... - AsynchronousEvents

2012-03-26T15:44:35Z

When I switched to looking at the performance of the more advanced RIO server designs that use IOCP it quickly became apparent that even multiple 1 Gigabit connections weren't enough of a challenge to give me any meaningful figures; my traditional IOCP datagram servers were easily able to keep up and increasing the workload per datagram required such high workloads that the tests became meaningless. So, we've brought forward the purchase of the hardware that we intended to use for our private cloud scalability testing and we now have 2 Intel 10 Gigabit AT2 cards. Switch prices are still prohibitive for lab use and so these two cards are directly connected, point to point.

The good news is that we now have a 10 Gigabit network link between two of our test servers. The bad news is that I now have to work out how to use it, the traditional datagram generation program that I was previously using to test simply doesn't scale to saturate the new link.

Windows 8 Registered I/O Performance - AsynchronousEvents

2012-03-15T15:15:00Z

I've been looking at the Windows 8 Registered I/O Networking Extensions since October when they first made an appearance as part of the Windows 8 Developer Preview. Whilst exploring and understanding the new API I spent some time putting together some simple UDP servers using the various notification styles that RIO provides. I then put together some equally simple UDP servers using the "traditional" APIs so that I could compare performance.

Of course these comparisons should be taken as preliminary since we're working with a beta version of the operating system. However, though I wouldn't put much weight in the exact numbers until we have a non-beta OS to test on, it's useful to see how things are coming along and familiarise ourselves with the designs that might be required to take advantage of RIO once it ships. The main thing to take away from these discussions on RIO's performance are the example server designs, the testing methods and a general understanding of why RIO performs better than the traditional Windows networking APIs. With this you can run your own tests, build your own servers and get value from using RIO where appropriate.

Do bear in mind that I'm learning as I go here, RIO is a new API and there is precious little in the way of documentation about how and why to use the API. Comments and suggestions are more than welcome, feel free to put me straight if I've made some mistakes, and submit code to help make these examples better.

How to test RIO's performance

The tests consist of sending a large number of datagrams to the server under test. We send two sizes of datagram, the test datagram and the shutdown datagram. The server counts the datagrams that it receives and the time taken. It shuts down as soon as it receives a shutdown datagram. The servers that we are using for these tests are detailed here and the datagram generator is available here.

Whilst the numbers that the servers report are useful for getting a rough idea of how the various API's compare they're not the whole story. It's useful to look at performance counter logs that are taken whilst the test server is running. The CPU usage of the server under test, and the entire machine, are useful indicators of how much further we could push a given server. The number of datagrams received, and dropped by the network and Winsock are useful to see, as is the non-paged pool usage, etc.

To make the testing repeatable I've put together some simple scripts which create the required performance logs using logman, the command line interface to perfmon. This means that for each test run we can run a single command which creates and starts a performance counter log, runs the server and then stops the performance counter log. It would be nice to include custom performance counters in each of the example servers so that we can see more of what's going on inside, but whilst easy to do, using our Performance Counters Option pack, that's beyond the scope of these tests.

The test client, or clients for when we're using two network links into the test machine, are started manually. We could automate this with winrs, as we've done in the past, but these tests don't really warrant that level of complexity.

Our test system

Our test system consists of a dual Xenon E5620 @ 2.40GHz, that's 16 CPUs in 2 Numa nodes with 16GB of memory. The machine has four 1Gb Ethernet network intefaces, a Broadcom BCM571C NetXtreme II GigE with two channels and a Intel 82576 Gigabit dual port adapter. We're using the Intel adapter for all of the tests shown here, sometimes using one NIC and sometimes two.

Windows Server 8 beta Datacentre edition is running directly on the hardware.

The client hardware is less impressive, but both client machines can push their 1Gb network interfaces to around 98% whilst running our datagram generator and that's more than enough for our purposes here.

The first tests

To get a feel for how the RIO API differs from the traditional API's the first test will compare a polled RIO server with a traditional, blocking, polled server. The code for the servers is available here along with some commentary on their designs. You'll need Visual Studio 11 to build the examples.

The test scripts, mentioned above, can be downloaded from here. Each server has its own script and a text file that details the performance counters to capture during the test run. All of the scripts call a common script which sets up the performance counter log and then starts the server. You shouldn't start the clients until the server is running and has output its configuration details. Once the server receives its first datagram it will display "TimingStarted" and when it has received a shutdown datagram it will display "TimingStopped" and display the number of datagrams that it managed to receive, the time taken and the datagrams per second. You need to copy the x64 release builds of the example servers into the same directory of the test scripts and then be sure to run the batch file and not the exe directly.

As an initial test we will run the traditional UDP server with one test client. We'll set the test client to send 10,000,000 datagrams, which takes a little over one minute. Once the test was completed the server reported that it had processed 9,952,510 datagrams in 86,880ms, a rate of 114,000 per second. Running the RIO polled server example with the same network load the results were broadly similar; 9,932,228 datagrams in 86,681ms, a rate of 114,000 per second.

At first glance it seems that RIO isn't so impressive, however we need to remind ourselves of what these example servers are doing; all they're doing is pulling datagrams off of the wire as fast as they can. They're both doing so on a single CPU of a 16 CPU machine and, from these results, it seems that on, this hardware, both APIs can quite easily handle a single saturated 1Gb network link.

Digging deeper into RIO's performance

Whilst the two servers at first appear to behave almost identically under the load it's only when we start looking at the performance counters that we can see that actually the two APIs have completely different performance characteristics.

Here's the graph for the traditional UDP server. Note the thick blue line, that's the amount of time the process spends in kernel mode, on average 37.133% of its time. The graph from the RIO server is a little different. The thick blue line is still there, it's just that it's 0 most of the time. The average is 0.167%. Another thing worth noting is that the spinning that the RIO server does is obvious from the fact that it uses 100% of a CPU (see the thick red line) and that most of that is spent in user mode code (the dotted green line that runs across the thick red line).

Another thing that is interesting to see is the non-paged pool usage; the RIO server uses a fixed amount for the life of the process, 8,064 bytes, the traditional server uses 4,192 bytes for most of the time but has some random peaks, the highest of which is 133,656 bytes.

Increasing the network load

Running two clients, each sending 10,000,000 datagrams to different network cards on the test machine gives us similar figures, the traditional server remains ahead of the RIO server with 19,847,578 datagrams to 19,279,842 datagrams. It seems that with the given hardware both APIs are capable of dealing with two saturated 1Gb links.

Increasing the work done per datagram received

The example servers all have a DoWork() function which allows us to add some "processing" for each datagram that is received, this gives us a slightly more realistic test as, except for discard servers, most servers need to do some work with each datagram that arrives.

Running the tests again, this time with a 'workload' of 100 gives the following results.

Traditional Server

6,386,294 datagrams out of 10,000,000 on 1 1Gb link, 63%
4,824,707 datagrams out of 20,000,000 on 2 1Gb links, 24%
38,830,887 datagrams out of 100,000,000 on 2 1Gb links, 38%

RIO Server

9,985,323 datagrams out of 10,000,000 on 1 1Gb link, 99%
19,730,003 datagrams out of 20,000,000 on 2 1Gb links, 98%
93,640,607 datagrams out of 100,000,000 on 2 1Gb links, 93%

Clearly a more realistic example allows the RIO API to show what it's capable of. Note that I ran a longer test, with two clients each sending 50,000,000 datagrams as the second test showed some results that seemed to imply that the traditional server had become overwhelmed near the end of the test. The longer test was to see if it could recover, it didn't it simply entered the overwhelmed state and stayed there until the end of the test. This is possibly due to the socket's recv buffer filling up.

As can be seen from these graphs, the traditional server quickly gets into a state where it is dropping vast numbers of datagrams (thick pink line) whilst burning more user mode CPU than kernel mode CPU having maxed out the single CPU that it's running on. The RIO server doesn't drop any datagrams and the graph looks surprisingly like the previous one with no load.

To look at how the performance of the RIO server degraded as the workload per datagram increases I ran some more tests.

RIO Server, 10,000,000 datagrams

9,985,323 datagrams, 99% at a workload of 100
9,888,733 datagrams, 98% at a workload of 300
7,174,653 datagrams, 71% at a workload of 500
5,573,046 datagrams, 55% at a workload of 700
4,361,820 datagrams, 43% at a workload of 1000
2,927,590 datagrams, 29% at a workload of 2000

And just to compare...
Traditional server, 10,000,000 datagrams

2,522,667 datagrams, 25% at a workload of 1000

Some conclusions

Bear in mind that these results are specific to the test machine I was running on and that we're testing on a beta version of the Windows 8 Server operating system. Even so, the figures are impressive. The lack of kernel mode transitions allow much more CPU to be used for real work on each datagram that arrives. The registering of I/O buffers once at program start up reduces the work done per operation and also means that your server will use a known amount of non-paged pool rather than a completely variable amount. Though non-paged pool is more plentiful than it used to be pre-Vista this is likely still an advantage.

The RIO API isn't especially complicated but your server designs will be different. The simple polling example server that we used here is unlikely to be an ideal choice as it uses 100% of its CPU for the whole time that the server is running. It's also a little unfair to compare RIO to such a simple traditional server but; there are better alternatives, but it's a useful line in the sand. As we'll see in the following performance articles there are better, and more scalable ways to use both APIs.

If you're interested in digging deeper into the results used in this article then all of the performance logs taken whilst running the tests are available here.

Security Clearance granted - Company News

2012-03-15T08:45:21Z

As we mentioned back in January, one of our clients has sponsored Len for SC level Security Clearance for one of the projects that we're bidding for with them.

We're pleased to announce that this has been granted and that Len now has SC level Security Clearance.

Whilst this clearance was obtained for work with our sponsoring client that work may or may not materialise, it's currently very early in the bidding process, so if you require the services of an Security Cleared C++ network specialist please get in touch.

Windows 8 Registered I/O - Generating load for the performance tests - AsynchronousEvents

2012-03-14T17:15:00Z

Now that we have five example servers, four RIO designs and a traditional polled UDP design, we can begin to look at how the RIO API performs compared to the traditional APIs. Of course these comparisons should be taken as preliminary since we're working with a beta version of the operating system. However, though I wouldn't put much weight in the exact numbers until we have a non-beta OS to test on, it's useful to see how things are coming along and familiarise ourselves with the designs that might be required to take advantage of RIO once it ships.

Sending a stream of datagrams

Before we can compare performance we need to be able to push the example servers hard. We do this by sending a stream of datagrams at them as fast as we can for a period of time. The servers start timing when they get the first datagram and then count the number of datagrams that they process. The test finishes by sending a series of smaller datagrams at the server. When the server sees one of these smaller datagrams it shuts down and reports on the time taken and the number of datagrams processed and the rate at which they were processed.

All we need to be able to do to stress the servers is to send datagrams at a rate that gets close to 100% utilisation of a 1Gb Ethernet link. This is fairly simple to achieve using the traditional blocking sockets API.

   for (size_t i = 0; i < DATAGRAMS_TO_SEND; ++i)
   {
      if (SOCKET_ERROR == ::WSASendTo(
         s,
         &buf,
         1,
         &bytesSent,
         flags,
         reinterpret_cast<sockaddr *>(&addr),
         sizeof(addr),
         0,
         0))
      {
         ErrorExit("WSASend");
      }
   }

There's not much more to it than that. We use similar code to setup and clean up, but if you've been following along with the other examples then there's nothing that needs to be explained about that.

The code for this example can be downloaded from here. This code requires Visual Studio 11, but would work with earlier compilers if you have a Windows SDK that supports RIO. Note that Shared.h and Constants.h contain helper functions and tuning constants for ALL of the examples and so there will be code in there that is not used by this example. You should be able to unzip each example into the same directory structure so that they all share the same shared headers. This allows you to tune all of the examples the same so that any performance comparisons make sense. This program can be run on versions of Windows prior to Windows 8, which is useful for testing as you only need one machine set up with the beta of Windows 8 server.

Windows 8 Registered I/O - Traditional Polled UDP Example Server - AsynchronousEvents

2012-03-14T14:00:00Z

This article presents the fifth in my series of example servers for comparing the performance of the Windows 8 Registered I/O Networking extensions, RIO, and traditional Windows networking APIs. This example server is a traditional polled UDP design that we can use to compare to the RIO polled UDP example server. I've been looking at the Windows 8 Registered I/O Networking Extensions since October when they first made an appearance as part of the Windows 8 Developer Preview. Whilst exploring and understanding the new API I spent some time putting together some simple UDP servers using the various notification styles that RIO provides. I then put together some equally simple UDP servers using the "traditional" APIs so that I could compare performance. This series of blog posts describes each of the example servers in turn. You can find an index to all of the articles about the Windows 8 Registered I/O example servers here.

A traditional polled UDP server

This server is probably the simplest UDP server you could have. It's pretty much just a tight loop around a blocking call to WSARecv(). There's none of the complexity required by RIO for registering memory buffers for I/O and so we use a single buffer that we create on the stack.

   do
   {
      workValue += DoWork(g_workIterations);

      if (SOCKET_ERROR == ::WSARecv(
         s,
         &buf,
         1,
         &bytesRecvd,
         &flags,
         0,
         0))
      {
         ErrorExit("WSARecv");
      }

      if (bytesRecvd == EXPECTED_DATA_SIZE)
      {
         g_packets++;
      }
      else
      {
         done = true;
      }
   }
   while (!done);

There is some added complexity to allow us to compare performance, and this is similar to the RIO server examples. We can add an arbitrary processing overhead to each datagram by setting g_workIterations to a non zero value and we count each datagram that arrives and stop the test when a datagram of an unexpected size is received.

Setting up for the datagram processing loop

As with the RIO examples we do some setup before we can process datagrams. See the polled RIO example server for details of how and why we set up the timing system and initialise Winsock, and for details on our error handling policy.

int _tmain(int argc, _TCHAR* argv[])
{
   SetupTiming("Simple polled UDP");

   InitialiseWinsock();

   SOCKET s = CreateSocket();

   Bind(s, PORT);

   SetSocketRecvBufferToMaximum(s);

   bool done = false;

   CHAR buffer[RECV_BUFFER_SIZE];

   WSABUF buf;

   buf.buf = buffer;
   buf.len = RECV_BUFFER_SIZE;

   DWORD bytesRecvd = 0;

   DWORD flags = 0;

   if (SOCKET_ERROR == ::WSARecv(
      s,
      &buf,
      1,
      &bytesRecvd,
      &flags,
      0,
      0))
   {
      ErrorExit("WSARecv");
   }

   g_packets++;

   StartTiming();

   int workValue = 0;

We then create a traditional blocking UDP socket, bind it to a port, set its receive buffer size to the maximum and create our receive buffer on the stack, set up our WSABUF and call WSARecv() for the first time. We make this call outside of our processing loop so that we can start timing when we get the first datagram. This code then proceeds into the processing loop, shown above, and processes datagrams until the test is complete and a datagram of an unexpected size is received.

The code for CreateSocket(), Bind() and SetSocketRecvBufferToMaximum() can be found in Shared.h and remember that the use of globals isn't clever, it's simply convenient for some of the other example servers that use the shared code.

After the processing loop

Once the performance test completes we stop our timing and report the results.

   StopTiming();

   PrintTimings();

   return workValue;
}

Join in

Comments and suggestions are more than welcome. I'm learning as I go here and I'm quite likely to have made some mistakes or come up with some erroneous conclusions, feel free to put me straight and help make these examples better.

Windows 8 Registered I/O - Multi threaded RIO IOCP UDP Example Server - AsynchronousEvents

2012-03-12T18:10:00Z

This article presents the fourth in my series of example servers using the Windows 8 Registered I/O Networking extensions, RIO. This example server, like the last example, uses the I/O Completion Port notification method to handle RIO completions, but where the last example used only a single thread to service the IOCP this one uses multiple thread to scale the load . I've been looking at the Windows 8 Registered I/O Networking Extensions since October when they first made an appearance as part of the Windows 8 Developer Preview. Whilst exploring and understanding the new API I spent some time putting together some simple UDP servers using the various notification styles that RIO provides. I then put together some equally simple UDP servers using the "traditional" APIs so that I could compare performance. This series of blog posts describes each of the example servers in turn. You can find an index to all of the articles about the Windows 8 Registered I/O example servers here.

Using an I/O Completion Port for RIO completions

As I mentioned back in October, there are three ways to receive completion notifications from RIO; polling, event driven and via an I/O Completion Port. Using an IOCP for RIO completions allows you to easily scale your completion handling across multiple threads as we do here and this is the first of my example servers that allows for more than one thread to be used to process completions.

Creating an IOCP driven RIO completion queue

We start by initialising things in the same way that we did with the earlier example RIO servers. In fact, this initialisation is identical to the previous IOCP example except for one thing.

int _tmain(int argc, _TCHAR* argv[])
{
   SetupTiming("RIO IOCP UDP");

   InitialiseWinsock();

   CreateRIOSocket();

   g_hIOCP = ::CreateIoCompletionPort(
      INVALID_HANDLE_VALUE,
      0,
      0,
      0);

   OVERLAPPED overlapped;

   RIO_NOTIFICATION_COMPLETION completionType;

   completionType.Type = RIO_IOCP_COMPLETION;
   completionType.Iocp.IocpHandle = g_hIOCP;
   completionType.Iocp.CompletionKey = (void*)1;
   completionType.Iocp.Overlapped = &overlapped;

   g_queue = g_rio.RIOCreateCompletionQueue(
      RIO_PENDING_RECVS,
      &completionType);

   if (g_queue == RIO_INVALID_CQ)
   {
      ErrorExit("RIOCreateCompletionQueue");
   }

With the previous design we passed 0 as the completion key. This server passes 1. This is an arbitrary change purely to allow us to post completions of 0 to cause all of the threads waiting on the completion queue to shut down. This is a common idiom with normal, non-RIO, IOCP designs as it's more usual that the completion key is a pointer to a "per operation" data structure. A RIO design with multiple completion queues would likely use the completion key for "per queue" data.

Creating a RIO request queue

Creating the request queue and posting our receives is identical to the polled example. The only difference is how we handle the completions.

Starting our worker threads

This example is the first to use more than just a single thread. Because of this we need to have a way to start and manage our worker threads and for them to communicate with each other and the main thread.

   CreateIOCPThreads(NUM_IOCP_THREADS);

   INT notifyResult = g_rio.RIONotify(g_queue);

   if (notifyResult != ERROR_SUCCESS)
   {
      ErrorExit("RIONotify", notifyResult);
   }

   WaitForProcessingStarted();

First we call CreateIOCPThreads(), which is shown below, this creates some events that the threads will use to communicate and then creates and starts the threads themselves. As with the earlier examples, we use globals for convenience and not as an example of good design.

inline void CreateIOCPThreads(
   const DWORD numThreads)
{
   g_hStartedEvent = ::CreateEvent(0, TRUE, FALSE, 0);

   if (0 == g_hStartedEvent)
   {
      ErrorExit("CreateEvent");
   }

   g_hStoppedEvent = ::CreateEvent(0, TRUE, FALSE, 0);

   if (0 == g_hStoppedEvent)
   {
      ErrorExit("CreateEvent");
   }

   // Start our worker threads

   for (DWORD i = 0; i < numThreads; ++i)
   {
      unsigned int notUsed;

      const uintptr_t result = ::_beginthreadex(
         0,
         0,
         ThreadFunction,
         0,
         0,
         &notUsed);

      if (result == 0)
      {
         ErrorExit("_beginthreadex", errno);
      }

      g_threads.push_back(reinterpret_cast<handle>(result));
   }

   cout << numThreads << " threads running" << endl;
}

The main thread then calls RIONotify() to enable notifications and then waits for the first datagram to be processed before it starts the timer.

inline void WaitForProcessingStarted()
{
   if (WAIT_OBJECT_0 != ::WaitForSingleObject(
      g_hStartedEvent,
      INFINITE))
   {
      ErrorExit("WaitForSingleObject");
   }

   StartTiming();
}

Calling RIODequeueCompletion() and processing results

This example's processing loop is similar to the previous examples, especially the single threaded IOCP example server. It's slightly more complicated due to the fact that it's being run on a separate thread.

unsigned int __stdcall ThreadFunction(
   void *pV)
{
   DWORD numberOfBytes = 0;

   ULONG_PTR completionKey = 0;

   OVERLAPPED *pOverlapped = 0;

   const DWORD recvFlags = 0;

   if (!::GetQueuedCompletionStatus(
      g_hIOCP,
      &numberOfBytes,
      &completionKey,
      &pOverlapped,
      INFINITE))
   {
      ErrorExit("GetQueuedCompletionStatus");
   }

   int workValue = 0;

   if (completionKey == 1)
   {
      RIORESULT results[RIO_MAX_RESULTS];

      bool done = false;

      ::SetEvent(g_hStartedEvent);

      ULONG numResults = g_rio.RIODequeueCompletion(
         g_queue,
         results,
         RIO_MAX_RESULTS);

      if (0 == numResults ||
          RIO_CORRUPT_CQ == numResults)
      {
         ErrorExit("RIODequeueCompletion");
      }

      INT notifyResult = g_rio.RIONotify(g_queue);

      if (notifyResult != ERROR_SUCCESS)
      {
         ErrorExit("RIONotify", notifyResult);
      }

      do
      {
         for (DWORD i = 0; i < numResults; ++i)
         {
            EXTENDED_RIO_BUF *pBuffer = reinterpret_cast<EXTENDED_RIO_BUF *>(results[i].RequestContext);

            if (results[i].BytesTransferred == EXPECTED_DATA_SIZE)
            {
               ::InterlockedIncrement(&g_packets);

               workValue += DoWork(g_workIterations);

               if (!g_rio.RIOReceive(
                  g_requestQueue,
                  pBuffer,
                  1,
                  recvFlags,
                  pBuffer))
               {
                  ErrorExit("RIOReceive");
               }

               done = false;
            }
            else
            {
               done = true;
            }
         }

         if (!done)
         {
            if (!::GetQueuedCompletionStatus(
               g_hIOCP,
               &numberOfBytes,
               &completionKey,
               &pOverlapped,
               INFINITE))
            {
               ErrorExit("GetQueuedCompletionStatus");
            }

            if (completionKey == 0)
            {
               done = true;
            }
            else
            {
               numResults = g_rio.RIODequeueCompletion(
                  g_queue,
                  results,
                  RIO_MAX_RESULTS);

               if (0 == numResults ||
                   RIO_CORRUPT_CQ == numResults)
               {
                  ErrorExit("RIODequeueCompletion");
               }

               INT notifyResult = g_rio.RIONotify(g_queue);

               if (notifyResult != ERROR_SUCCESS)
               {
                  ErrorExit("RIONotify", notifyResult);
               }
            }
         }
      }
      while (!done);
   }

   ::SetEvent(g_hStoppedEvent);

   return workValue;
}

The first thing we do is wait for a completion. Once we have a completion we dequeue the results and then call RIONotify() to allow more completions to occur. It's important to realise that until we call RIONotify() no further completions will be posted to the I/O Completion Port and that this effectively acts as synchronisation around the calls to RIODequeueCompletion(). With this design only one thread can ever be calling RIODequeueCompletion() at a time, which is a good thing as the documentation for RIODequeueCompletion() states that this is a requirement for users of the API.

Remember that this example is made more complex due to the way we profile the servers. See the explanation in the completion handling section of the polled RIO server example for details of why this is.

Shutting down and displaying results

Whilst our worker threads are processing datagrams our main thread is waiting for the performance test to end.

   WaitForProcessingStopped();

   StopIOCPThreads();

   PrintTimings();

   return 0;
}

Our thread function's main loop can exit in two ways. Firstly because a datagram arrives that isn't of the expected size, thus signalling the end of the performance test, and secondly if GetQueuedCompletionStatus() returns a completion key of 0 which means that the main thread has posted completions to request that we shut down. This means that the first time a "shutdown" datagram arrives the first thread that begins processing it will shut down and set the g_hStoppedEvent event. The main thread is currently waiting for this event, and will wake when the event is set and shut the rest of the worker threads down. Once all of the threads have terminated the main thread will display details of the datagrams received and the test timings.

inline void WaitForProcessingStopped()
{
   if (WAIT_OBJECT_0 != ::WaitForSingleObject(
      g_hStoppedEvent,
      INFINITE))
   {
      ErrorExit("WaitForSingleObject");
   }

   StopTiming();
}

inline void StopIOCPThreads()
{
   // Tell all threads to exit

   for (Threads::const_iterator it = g_threads.begin(),
      end = g_threads.end();
      it != end;
      ++it)
   {
      if (0 == ::PostQueuedCompletionStatus(
         g_hIOCP,
         0,
         0,
         0))
      {
         ErrorExit("PostQueuedCompletionStatus");
      }
   }

   cout << "Threads stopping" << endl;

   // Wait for all threads to exit

   for (Threads::const_iterator it = g_threads.begin(),
      end = g_threads.end();
      it != end;
      ++it)
   {
      HANDLE hThread = *it;

      if (WAIT_OBJECT_0 != ::WaitForSingleObject(
         hThread,
         INFINITE))
      {
         ErrorExit("WaitForSingleObject");
      }

      ::CloseHandle(hThread);
   }   

   cout << "Threads stopped" << endl;
}

Unexpected performance issues...

The slight problem with this design is that it's not actually as performant in some scenarios as we might like it to be. As it stands, the fact that we can scale out across multiple threads is a plus point but the fact that the operations that we have to perform to achieve that scaling are considerably more expensive is a problem. This is more of an issue when we're looking for a general purpose solution which works as well for low throughput and fast processing of each datagram as it does for high throughput and/or slow processing. Luckily there are a couple of things we can do to fix this, but we'll look at those once we've done some performance comparisons and seen the problems first hand.

Join in

Windows 8 Registered I/O - Single threaded RIO IOCP UDP Example Server - AsynchronousEvents

2012-03-12T15:45:00Z

This article presents the third in my series of example servers using the Windows 8 Registered I/O Networking extensions, RIO. This example server uses the I/O Completion Port notification method to handle RIO completions, but only uses a single thread to service the IOCP. I've been looking at the Windows 8 Registered I/O Networking Extensions since October when they first made an appearance as part of the Windows 8 Developer Preview. Whilst exploring and understanding the new API I spent some time putting together some simple UDP servers using the various notification styles that RIO provides. I then put together some equally simple UDP servers using the "traditional" APIs so that I could compare performance. This series of blog posts describes each of the example servers in turn. You can find an index to all of the articles about the Windows 8 Registered I/O example servers here.

Using an I/O Completion Port for RIO completions

As I mentioned back in October, there are three ways to receive completion notifications from RIO; polling, event driven and via an I/O Completion Port. Using an IOCP for RIO completions allows you to easily scale your completion handling across multiple threads, though in this first IOCP example server we use a single thread so as to allow us to compare the performance against the polled and event driven servers. The next example server will adapt this server for multiple threads and allow us to scale our completion processing across more CPUs.

Creating an IOCP driven RIO completion queue

We start by initialising things in the same way that we did with the earlier example RIO servers.

int _tmain(int argc, _TCHAR* argv[])
{
   SetupTiming("RIO IOCP UDP");

   InitialiseWinsock();

   CreateRIOSocket();

   g_hIOCP = ::CreateIoCompletionPort(
      INVALID_HANDLE_VALUE,
      0,
      0,
      0);

   OVERLAPPED overlapped;

   RIO_NOTIFICATION_COMPLETION completionType;

   completionType.Type = RIO_IOCP_COMPLETION;
   completionType.Iocp.IocpHandle = g_hIOCP;
   completionType.Iocp.CompletionKey = (void*)0;
   completionType.Iocp.Overlapped = &overlapped;

   g_queue = g_rio.RIOCreateCompletionQueue(
      RIO_PENDING_RECVS,
      &completionType);

   if (g_queue == RIO_INVALID_CQ)
   {
      ErrorExit("RIOCreateCompletionQueue");
   }

Once that is done we create an IO Completion Port and then create a RIO completion queue which uses the IOCP for notification. In this simple design we have no need for a completion key as we only have a single completion queue so there's no need to differentiate between completion types. We also use a plain old OVERLAPPED rather than extending it to carry more information. More complex designs could use either the completion key, or an extended overlapped structure to pass queue specific information to our completion handler in much the same way that we do with normal IOCP server designs.

Creating a RIO request queue

Creating the request queue and posting our receives is identical to the polled example. The only difference is how we handle the completions.

Calling RIODequeueCompletion() and processing results

Processing completions is almost identical to processing event driven completions. We simply change the call to WaitForSingleObject() that we were using in the event driven example to the following to retrieve a completion notification from the IOCP.

   DWORD numberOfBytes = 0;

   ULONG_PTR completionKey = 0;

   OVERLAPPED *pOverlapped = 0;

   if (!::GetQueuedCompletionStatus(
      g_hIOCP,
      &numberOfBytes,
      &completionKey,
      &pOverlapped,
      INFINITE))
   {
      ErrorExit("GetQueuedCompletionStatus");
   }

Everything else is identical. Things change somewhat when we switch to using multiple threads for our completion handling.

Join in

Windows 8 Registered I/O - Single threaded RIO Event Driven UDP Example Server - AsynchronousEvents

2012-03-09T22:25:00Z

This article presents the second in my series of example servers using the Windows 8 Registered I/O Networking extensions, RIO. This example server uses the event driven notification method to handle RIO completions. I've been looking at the Windows 8 Registered I/O Networking Extensions since October when they first made an appearance as part of the Windows 8 Developer Preview. Whilst exploring and understanding the new API I spent some time putting together some simple UDP servers using the various notification styles that RIO provides. I then put together some equally simple UDP servers using the "traditional" APIs so that I could compare performance. This series of blog posts describes each of the example servers in turn. You can find an index to all of the articles about the Windows 8 Registered I/O example servers here.

Using an event for RIO completions

As I mentioned back in October, there are three ways to receive completion notifications from RIO; polling, event driven and via an I/O Completion Port. Using the event driven approach is similar to using the polling approach that I described in the previous article except that the server doesn't burn CPU in a tight polling loop.

Creating an event driven RIO completion queue

We start by initialising things in the same way that we did with the earlier example RIO servers.

int _tmain(int argc, _TCHAR* argv[])
{
   SetupTiming("RIO Event Driven UDP");

   InitialiseWinsock();

   CreateRIOSocket();

   HANDLE hEvent = WSACreateEvent();

   if (hEvent == WSA_INVALID_EVENT)
   {
      ErrorExit("WSACreateEvent");
   }

   RIO_NOTIFICATION_COMPLETION completionType;

   completionType.Type = RIO_EVENT_COMPLETION;
   completionType.Event.EventHandle = hEvent;
   completionType.Event.NotifyReset = TRUE;

   g_queue = g_rio.RIOCreateCompletionQueue(
      RIO_PENDING_RECVS,
      &completionType);

   if (g_queue == RIO_INVALID_CQ)
   {
      ErrorExit("RIOCreateCompletionQueue");
   }

Once that is done we create an event and then create a RIO completion queue which uses the event for notification. The event is signalled when there are completions to process and reset when we call RIONotify().

Creating a RIO request queue

Creating the request queue and posting our receives is identical to the polled example. The only difference is how we handle the completions.

Calling RIODequeueCompletion() and processing results

The processing loop is, again, similar to the polled example. Unsurprisingly rather than polling we wait on the event and dequeue the completions once the event is set. This reduced the amount of CPU used as there's no need to spin whilst waiting for new datagrams to process. The only complication is that we need to call RIONotify() to indicate that we're ready to process more completions. Note that in a real server you would probably want to wait on your completions available event and a 'we're ready to shut down' event so that you can shut the sever down cleanly.

   bool done = false;

   DWORD recvFlags = 0;

   RIORESULT results[RIO_MAX_RESULTS];

   const INT notifyResult = g_rio.RIONotify(g_queue);

   if (notifyResult != ERROR_SUCCESS)
   {
      ErrorExit("RIONotify");
   }

   const DWORD waitResult = WaitForSingleObject(
      hEvent,
      INFINITE);

   if (waitResult != WAIT_OBJECT_0)
   {
      ErrorExit("WaitForSingleObject");
   }

   ULONG numResults = g_rio.RIODequeueCompletion(
      g_queue,
      results,
      RIO_MAX_RESULTS);

   if (0 == numResults ||
       RIO_CORRUPT_CQ == numResults)
   {
      ErrorExit("RIODequeueCompletion");
   }

   StartTiming();

   int workValue = 0;

   bool running = true;

   do
   {
      for (DWORD i = 0; i < numResults; ++i)
      {
         EXTENDED_RIO_BUF *pBuffer = reinterpret_cast<EXTENDED_RIO_BUF *>(results[i].RequestContext);

         if (results[i].BytesTransferred == EXPECTED_DATA_SIZE)
         {
            g_packets++;

            workValue += DoWork(g_workIterations);

            if (!g_rio.RIOReceive(
               g_requestQueue,
               pBuffer,
               1,
               recvFlags,
               pBuffer))
            {
               ErrorExit("RIOReceive");
            }

            done = false;
         }
         else
         {
            done = true;
         }
      }

      if (!done)
      {
         const INT notifyResult = g_rio.RIONotify(g_queue);

         if (notifyResult != ERROR_SUCCESS)
         {
            ErrorExit("RIONotify");
         }

         const DWORD waitResult = WaitForSingleObject(
            hEvent,
            INFINITE);

         if (waitResult != WAIT_OBJECT_0)
         {
            ErrorExit("WaitForSingleObject");
         }

         numResults = g_rio.RIODequeueCompletion(
            g_queue,
            results,
            RIO_MAX_RESULTS);

         if (0 == numResults ||
             RIO_CORRUPT_CQ == numResults)
         {
            ErrorExit("RIODequeueCompletion");
         }
      }
   }
   while (!done);

   StopTiming();

   PrintTimings();

   return workValue;
}

As before, the structure of the processing loop is complicated somewhat by the fact that we want to start and stop the timing for the performance testing, and the DoWork() function can be used to add 'processing overhead' to each datagram. This can be configured using the g_workIterations which is defined in Constants.h. With this set to 0 there is no overhead and we can compare how quickly each API can receive datagrams. Setting larger values will affect how the various multi-threaded examples perform and can be useful if you're unable to saturate the test machine's network interfaces.

This example can be optimised slightly so that we revert to straight polling as long as calling RIODequeueCompletion() returns us at least one result. We'll look at this variation after we've studied the performance of the example shown here.

Join in

Visual Studio 11, the UI changes don't matter... - Rambling Comments

2012-03-09T21:50:40Z

The best thing about Visual Studio 11 is that it doesn't matter if you like the new style IDE or not. The project files are, at last, backwards compatible, so you can load them in Visual Studio 2010 and build with the new tool chain even though you ignore the new IDE - if that's what you want to do.

I don't like the new icons, but I find I can work fine in the IDE as long as I don't think about it too much... Probably pretty much like how I felt about all previous versions when they were at the beta stage...

Windows 8 Registered I/O - Simple RIO Polled UDP Example Server - AsynchronousEvents

2012-03-07T11:28:00Z

Polling RIO for completions

As I mentioned back in October, there are three ways to receive completion notifications from RIO; polling, event driven and via an I/O Completion Port. The first is the simplest though it burns CPU time even when no datagrams are being received.

At its simplest a polled RIO server obtains datagrams to process like this:

   RIORESULT results[RIO_MAX_RESULTS];

   ULONG numResults = 0;

   do
   {
      numResults = g_rio.RIODequeueCompletion(
         g_queue,
         results,
         RIO_MAX_RESULTS);

      if (0 == numResults)
      {
         YieldProcessor();
      }
      else if (RIO_CORRUPT_CQ == numResults)
      {
         ErrorExit("RIODequeueCompletion");
      }
   }
   while (0 == numResults);

You then loop over the results array and process each result in turn before looping back to dequeue more completions.

Getting to the point where you can call RIODequeueCompletion() takes a bit of setting up though...

Creating a RIO completion queue

The examples are each stand alone but can share two common header files. The first, Constants.h, contains all constants that are used to tune the examples. The second, Shared.h, contains inline helper functions which hide some of the complexity and allow the individual example programs to focus on the area of the API that they're demonstrating. We use several of these helper functions as we prepare to create our RIO completion queue.

int _tmain(int argc, _TCHAR* argv[])
{
   SetupTiming("RIO polled UDP");

   InitialiseWinsock();

   CreateRIOSocket();

   g_queue = g_rio.RIOCreateCompletionQueue(RIO_PENDING_RECVS, 0);

   if (g_queue == RIO_INVALID_CQ)
   {
      ErrorExit("RIOCreateCompletionQueue");
   }

So that we can compare the performance of these examples we first call SetupTiming(), this prepares us for calling StartTiming() and StopTiming() later in the program. SetupTiming() locks this thread to a single CPU, using SetThreadAffinityMask() as recommended by the help for QueryPerformanceCounter(), once this is done we call QueryPerformanceFrequency() and store the resulting value for use by StopTiming().

With that done we initialise Winsock and then create our RIO socket. Note that these examples use quite a few global variables for convenience. This isn't how I would suggest you write production code, but it's convenient for these examples as it makes much of the code simpler and allows us to focus on the RIO API. For example, CreateRIOSocket() does the following, creating a socket and assigning it to g_s, we then bind the socket to PORT, which is a constant that is defined in Constants.h, and then initialising the RIO API function table so that we can use it through g_rio.

inline void CreateRIOSocket()
{
   g_s = CreateSocket(WSA_FLAG_REGISTERED_IO);

   Bind(g_s, PORT);

   InitialiseRIO(g_s);
}

inline SOCKET CreateSocket(
   const DWORD flags = 0)
{
   g_s = ::WSASocket(
      AF_INET,
      SOCK_DGRAM,
      IPPROTO_UDP,
      NULL,
      0,
      flags);

   if (g_s == INVALID_SOCKET)
   {
      ErrorExit("WSASocket");
   }

   return g_s;
}

inline void InitialiseRIO(
   SOCKET s)
{
   GUID functionTableId = WSAID_MULTIPLE_RIO;

   DWORD dwBytes = 0;

   bool ok = true;

   if (0 != WSAIoctl(
      s,
      SIO_GET_MULTIPLE_EXTENSION_FUNCTION_POINTER,
      &functionTableId,
      sizeof(GUID),
      (void**)&g_rio,
      sizeof(g_rio),
      &dwBytes,
      0,
      0))
   {
      ErrorExit("WSAIoctl");
   }
}

As you can see, we check all API calls for failure and report errors via our ErrorExit() functions.

Finally we can create the RIO completion queue. We use the tunable constant RIO_PENDING_RECVS to specify how large the queue should be.

Creating a RIO request queue

Setting up a RIO request queue seems to be one of the few places where RIO API changes between the Windows 8 Developer Preview and the Windows 8 Server Beta are visible to us. With the Developer Preview we could pass any value for the maxReceiveDataBuffers and maxSendDataBuffers whereas in the beta RIOCreateRequestQueue() only accepts a value of 1. See my earlier posting on RIO for more details, but the documentation and the API are now in sync and the API currently doesn't support scatter/gather I/O.

   ULONG maxOutstandingReceive = RIO_PENDING_RECVS;
   ULONG maxReceiveDataBuffers = 1;
   ULONG maxOutstandingSend = 0;
   ULONG maxSendDataBuffers = 1;

   void *pContext = 0;

   g_requestQueue = g_rio.RIOCreateRequestQueue(
      g_s,
      maxOutstandingReceive,
      maxReceiveDataBuffers,
      maxOutstandingSend,
      maxSendDataBuffers,
      g_queue,
      g_queue,
      pContext);

   if (g_requestQueue == RIO_INVALID_RQ)
   {
      ErrorExit("RIOCreateRequestQueue");
   }

   PostRIORecvs(RECV_BUFFER_SIZE, RIO_PENDING_RECVS);

Once the request queue has been created we can post some read requests. Note that we specify the size of the buffers to use and the number of receives that we want to have pending, both of these values can be changed easily in Constants.h.

Registering buffers and posting RIO read requests

Before we can issue some read requests we need to register some I/O buffers. The PostRIORecvs() function is complicated by the fact that we're trying to keep things simple (!) and by the fact that the example is tunable for different buffer sizes and the number of receives that we may want pending. We loop allocating and registering buffers and then slicing the buffer into buffer slices. We use an extended RIO_BUF structure so that we can pass an "operation" code with each buffer slice. These examples don't use this operation, but a real server might need to pass additional information with each I/O request, especially if it's using a single completion queue for reads and writes. We deliberately leak our EXTENDED_RIO_BUF structures but this isn't much of a problem in this example as they're in use from this point until the program exits.

inline void PostRIORecvs(
   const DWORD recvBufferSize,
   const DWORD pendingRecvs)
{
   DWORD totalBuffersAllocated = 0;

   while (totalBuffersAllocated < pendingRecvs)
   {
      DWORD bufferSize = 0;
   
      DWORD receiveBuffersAllocated = 0;

      char *pBuffer = AllocateBufferSpace(
         recvBufferSize,
         pendingRecvs,
         bufferSize,
         receiveBuffersAllocated);

      totalBuffersAllocated += receiveBuffersAllocated;

      RIO_BUFFERID id = g_rio.RIORegisterBuffer(
         pBuffer,
         static_cast<DWORD>(bufferSize));

      if (id == RIO_INVALID_BUFFERID)
      {
         ErrorExit("RIORegisterBuffer");
      }

      DWORD offset = 0;

      const DWORD recvFlags = 0;

      EXTENDED_RIO_BUF *pBufs = new EXTENDED_RIO_BUF[receiveBuffersAllocated];

      for (DWORD i = 0; i < receiveBuffersAllocated; ++i)
      {
         // now split into buffer slices and post our recvs

         EXTENDED_RIO_BUF *pBuffer = pBufs + i;

         pBuffer->operation = 0;
         pBuffer->BufferId = id;
         pBuffer->Offset = offset;
         pBuffer->Length = recvBufferSize;

         offset += recvBufferSize;

         if (!g_rio.RIOReceive(g_requestQueue, pBuffer, 1, recvFlags, pBuffer))
         {
            ErrorExit("RIOReceive");
         }
      }

      if (totalBuffersAllocated != pendingRecvs)
      {
         cout << pendingRecvs << " receives pending" << endl;
      }
   }

   cout << totalBuffersAllocated << " total receives pending" << endl;
}

Before we can register our I/O buffer we need to allocate the memory that we will be using. As I mentioned back in October when I was first looking at the RIO API, it's important to allocate your I/O buffer memory in a particular way so that you use memory efficiently for RIO's registered I/O buffers.

inline char *AllocateBufferSpace(
   const DWORD recvBufferSize,
   const DWORD pendingRecvs,
   DWORD &bufferSize,
   DWORD &receiveBuffersAllocated)
{
   const DWORD preferredNumaNode = 0;

   const SIZE_T largePageMinimum = USE_LARGE_PAGES ? ::GetLargePageMinimum() : 0;

   SYSTEM_INFO systemInfo;

   ::GetSystemInfo(&systemInfo);

   systemInfo.dwAllocationGranularity;
   
   const unsigned __int64 granularity = (largePageMinimum == 0 ? systemInfo.dwAllocationGranularity : largePageMinimum);

   const unsigned __int64 desiredSize = recvBufferSize * pendingRecvs;

   unsigned __int64 actualSize = RoundUp(desiredSize, granularity);

   if (actualSize > std::numeric_limits<DWORD>::max())
   {
      actualSize = (std::numeric_limits<DWORD>::max() / granularity) * granularity;
   }

   receiveBuffersAllocated = std::min<DWORD>(pendingRecvs, static_cast<DWORD>(actualSize / recvBufferSize));

   bufferSize = static_cast<DWORD>(actualSize);

   char *pBuffer = reinterpret_cast<char *>(VirtualAllocExNuma(
      GetCurrentProcess(),
      0,
      bufferSize,
      MEM_COMMIT |
      MEM_RESERVE  |
      (largePageMinimum != 0 ? MEM_LARGE_PAGES : 0),
      PAGE_READWRITE,
      preferredNumaNode));

   if (pBuffer == 0)
   {
      ErrorExit("VirtualAlloc");
   }

   return pBuffer;
}

Our allocation function is again slightly more complex than it need be, but that complexity allows us to explore various options by simply changing our configuration constants; you can ignore things like the USE_LARGE_PAGES flag and the fact that we're allocating to a preferred NUMA node unless you're interested in the details and your hardware supports these features. The important thing is that we allocate in terms of the system's allocation granularity and that we use a variant of VirtualAlloc() to do. Once again, in the name of simplicity, we leak these buffers (which will be in use for the life of the program) and allow program exit to clean them up.

Calling RIODequeueCompletion() and processing results

We finally have queues created, buffers registered and reads pending. Processing these reads in our simple polled RIO server is fairly straight forward. First we enter a polling loop for completions and spin until completions are available. Once we have at least one completion we call StartTiming() to start our performance timing. We then process the completion results. Our performance tests are simple, we send a number of datagrams of EXPECTED_DATA_SIZE and then indicate that the test is complete by sending a series of datagrams of a different size. Once our servers receive an unexpected sized datagram they consider the test to be complete and shutdown. Thus our main completion loop is the do/while loop below. We process datagrams, issue new reads and then dequeue more results. Once we're done we stop our timer and display details about the time taken and the number of datagrams that we processed.

bool done = false;

   DWORD recvFlags = 0;

   RIORESULT results[RIO_MAX_RESULTS];

   ULONG numResults = 0;

   do
   {
      numResults = g_rio.RIODequeueCompletion(
         g_queue,
         results,
         RIO_MAX_RESULTS);

      if (0 == numResults)
      {
         YieldProcessor();
      }
      else if (RIO_CORRUPT_CQ == numResults)
      {
         ErrorExit("RIODequeueCompletion");
      }
   }
   while (0 == numResults);

   StartTiming();

   int workValue = 0;

   bool running = true;

   do
   {
      for (DWORD i = 0; i < numResults; ++i)
      {
         EXTENDED_RIO_BUF *pBuffer = reinterpret_cast<EXTENDED_RIO_BUF *>(results[i].RequestContext);

         if (results[i].BytesTransferred == EXPECTED_DATA_SIZE)
         {
            g_packets++;

            workValue += DoWork(g_workIterations);

            if (!g_rio.RIOReceive(
               g_requestQueue,
               pBuffer,
               1,
               recvFlags,
               pBuffer))
            {
               ErrorExit("RIOReceive");
            }

            done = false;
         }
         else
         {
            done = true;
         }
      }

      if (!done)
      {
         do
         {
            numResults = g_rio.RIODequeueCompletion(
               g_queue,
               results,
               RIO_MAX_RESULTS);

            if (0 == numResults)
            {
               YieldProcessor();
            }
            else if (RIO_CORRUPT_CQ == numResults)
            {
               ErrorExit("RIODequeueCompletion");
            }
         }
         while (0 == numResults);
      }
   }
   while (!done);

   StopTiming();

   PrintTimings();

   return workValue;
}

The DoWork() function above can be used to add 'processing overhead' to each datagram. This can be configured using the g_workIterations which is defined in Constants.h. With this set to 0 there is no overhead and we can compare how quickly each API can receive datagrams. Setting larger values will affect how the various multi-threaded examples perform and can be useful if you're unable to saturate the test machine's network interfaces.

Join in

Windows 8 Registered I/O Example UDP Servers - AsynchronousEvents

2012-03-07T11:27:00Z

RIO API demonstration

The examples are simple in that they do the bare minimum to demonstrate the APIs in question but they are configurable so that you can tune them to the hardware on which you're running them. You can run them to compare the maximum speed at which you can pull UDP datagrams off of the wire using each API and then adjust the examples so that they do a specific amount of "work" with each datagram to simulate a slightly more realistic scenario.

Simplified error handling

Error handling is limited, we display an error and exit the program, but we don't skip error checking, all API calls are checked for errors. The examples are each stand alone but can share two common header files. The first, Constants.h, contains all constants that are used to tune the examples. The second, Shared.h, contains inline helper functions which hide some of the complexity and allow the individual example programs to focus on the area of the API that they're demonstrating.

This is the index page

I will be blogging about the construction of the various examples over the next few weeks and updating this entry as an index page for all of the examples. I've listed the examples that I'll be talking about and I'll link to each blog post as they go live. Once I've presented the RIO examples I'll present the more traditional examples and finally some performance comparisons.

RIO server examples

RIO Polled UDP - A server which uses a single thread and a tight loop to poll for RIO completions.
RIO Event Driven UDP - A server which uses a single thread and event driven notifications to handle RIO completions.
RIO IOCP UDP - A server which uses a single thread and I/O Completion Port notifications to handle RIO completions.
RIO IOCP MT UDP - A server which uses a configurable number of threads and I/O Completion Port notifications to handle RIO completions.

Traditional server examples

Simple Polled UDP - A server which uses a single thread and a tight loop to poll WSARecv() for datagrams.
IOCP UDP - A server which uses a single thread and I/O Completion Port notifications to handle overlapped WSARecv() completions.
IOCP MT UDP - A server which uses a configurable number of threads and I/O Completion Port notifications to handle overlapped WSARecv() completions.

A simple UDP datagram traffic generator

Simple UDP traffic generator - A client which uses a single thread and a tight loop send datagrams using WSASendTo(), this easily saturates a 1000BASE-T, 1Gb ethernet connection.

Test scripts

Test scripts - These simple scripts create performance counter logs and run the test servers.

Performance Test results

The first tests - Where we compare the simple polled traditional server with the polled RIO server.

Join in

New release of deadlock detection tools - Lock Explorer

2012-02-20T14:49:07Z

We've released new versions of both LID and LIA today. These releases include many changes that we've been testing with customers over the last couple of months and mainly deal with fixing hangs during the shutdown of managed applications when they are run under the tools and performance improvements when running on target processes which use a large number of locks and create a large number of lock acquisition sequences.

In addition we've fixed some bugs with TryEnterCriticalSection() where we were incorrectly reporting lock inversions when a call to TryEnterCriticalSection() returned FALSE and added a few extra command line switches so that you can show progress during lock inversion detection after the target process completes.

You can download the latest version of LID from here and all customers are being contacted via email with details of how to download the latest release of LIA.

Do continue to get in touch with comments and suggestions and any problems that you have.

Latest release of The Server Framework: 6.5.4 - AsynchronousEvents

2012-02-14T08:28:41Z

Version 6.5.4 of The Server Framework was released today.

This release contains two important bug fixes and a selection of minor improvements. If you run your code on Vista/Windows Server 2003 or later and you don't explicitly disable FILE_SKIP_COMPLETION_PORT_ON_SUCCESS in your Config.h then you should install this update.

This release includes the following, see the release notes, here, for full details of all changes.

Bug fix. If FILE_SKIP_COMPLETION_PORT_ON_SUCCESS was enabled but JetByteTools::Socket::CanEnableSkipCompletionPortOnSuccess() returned false then the the code that handled issuing read and write calls would fail if ERROR_SUCCESS was returned because it would assume that FILE_SKIP_COMPLETION_PORT_ON_SUCCESS was enabled and that it should handle the completion directly but a completion would have been posted to the IOCP and so the completion would get handled twice. We now correctly whether we have actually enabled FILE_SKIP_COMPLETION_PORT_ON_SUCCESS rather than just whether we want to enable it.
Change to JetByteTools::Socket::CConnectionMaintainingStreamSocketConnectionFilter so that we do not attempt to maintain a connection if the reconnect delay is 0.
Added the concept of being able to force a write request to go via the I/O pool even if marshalling is currently turned off.
Added JetByteTools::Socket::IManageStreamSocketConnectionFilters::TryRequestWrite()
Changed how JetByteTools::Socket::CFlowControlStreamSocketConnectionFilter issues write requests and how it deals with write failure due to socket closure. We now purge any queued data when we detect the socket has been closed, rather than continuing to try and send more.
Added JetByteTools::Socket::IDatagramSendSocket which is a common base class for JetByteTools::Socket::IDatagramSocket and JetByteTools::Socket::IDatagramServerSocket.
Added JetByteTools::Socket::IFilterableStreamSocket::CanIssueFilteredWrite() which is now called instead of JetByteTools::Socket::IStreamSocketEx::CanWrite() by JetByteTools::Socket::CFilteringStreamSocketConnectionManagerBase::TryRequestWrite(). This removes a race condition during the shutdown of the write side of a socket in situations where filtering is being used and the filter wishes to write to the socket after the application level code has requested that the write side of the socket be shut down. We always tracked the outstanding write count before actually issuing the shutdown and the filter could manage this to allow it to be able to send after a shutdown had been requested BUT the filtered send could still fail as the socket's write shutdown flag would be set. This new function does not check the write shutdown flag and so allows the filter to write successfully.
Changed JetByteTools::IO::IAllocateBufferHandles::Flush() so that it returns a bool indicating if buffers were active when the flush was done. This brings it in line with JetByteTools::IO::IAllocateBuffers.
Changed the JetByteTools::IO::IAsyncIOStream::Write() methods so that they take an optional bool that enables you to force the write to go via the I/O pool even if I/O marshalling is turned off.
Changed JetByteTools::IO::CAsyncFileLog to monitor its own write thread to remove the chance that it might hang during destruction if the thread has terminated due to an exception.

The advantage of having lots of clients and clients with lots of clients - AsynchronousEvents

2012-02-08T12:11:02Z

Our Secretive Online Game Company client uses The Server Framework for their custom application server for the games industry. They have thousands of users who run their server on a very diverse set of hardware. This is great for us as it really helps to shake down The Server Framework. There's nothing like running your multi-threaded code on lots of different hardware to help find all of the hidden race conditions and whatever. I'm pleased that we have so few bug reports coming in from our clients. Especially knowing that our Online Game Company client has the latest code out in the field or at least in use internally on their cloud system. Unfortunately one of their clients has recently exposed a latent bug in a rarely used corner of The Server Framework.

One of the features of The Server Framework is that we track new features in various Windows operating systems so that you can take advantage of them simply by upgrading. Often you only need to adjust your Config.h file to enable powerful new features. One of these features is FILE_SKIP_COMPLETION_PORT_ON_SUCCESS, you can read more about it over on Len's blog. This allows some optimisation in thread scheduling and context switching and generally improves performance on busy servers.

Back in April 2011 we updated our FILE_SKIP_COMPLETION_PORT_ON_SUCCESS support to include protection from incompatible networking providers, see this Microsoft Knowledge Base article for details of the potential problem. In addition to the compile time support for FILE_SKIP_COMPLETION_PORT_ON_SUCCESS which can be used to turn the feature on and off in The Server Framework, we added a run time check to ensure that the machine on which the code was running did not have any incompatible networking providers installed.

Unfortunately this code wasn't tested as well as it could be and there was a bug in it which leads to problems on systems that have an incompatible networking provider installed and FILE_SKIP_COMPLETION_PORT_ON_SUCCESS support turned on. This leads to the completions for some operations being processed twice and subsequent reference counting (over-release) problems with the corresponding socket and buffer structures.

We've fixed this issue and it will be included in a 6.5.4 release which is currently in test. If you think you're suffering from the problems caused by this and need the fix immediately then please get in touch.

The offending networking provider in this case was the "AVSDA" provider which, a quick web search suggests, is part of Avira Anti virus.

WASP download of XP versions now fixed - AsynchronousEvents

2012-02-02T14:58:40Z

I've just noticed a problem with downloading the XP versions of WASP.

This is now fixed. The XP versions can now be downloaded correctly again from here. Sorry for any inconvenience caused.

Happy New Year - Company News

2012-01-13T15:42:57Z

The year has kicked off to a very busy start for us with lots of work from our secretive Online Gaming Company. They're doing a lot of work to enhance the product that we've helped them build so that it can run well in cloud environments for their clients and also form the core of their cloud-based service. Much of our current work for them is to do with server to server communications so that they can build a scalable system that can use resources in the cloud to grow on demand. We've been working with them on aspects of this for some time now and it's all starting to come together nicely. Additionally we've recently added WebSockets support to their server so that they can easily support a new breed of browser based clients.

We're also excited about working more closely with our Industrial Control Client who are sponsoring us for SC level security clearance so that we can get more involved with their systems; if we told you any more we'd have to kill you...

We've had more fixed price work for our Custom Electronics Manufacturer, adding SChannel based SSL support to their server. This server allows thousands of embedded devices to connect to their central servers to upload logs and download firmware.

We've still been too busy to do much work on the LockExplorer website but there's an update to the tools scheduled for the end of the month to include all of the recent changes and fixes that have been applied whilst we've been 'dog fooding' them with our clients and on The Server Framework's build machines.

Finally, we already have several releases planned for The Server Framework this year; watch this space!

Latest release of The Server Framework: 6.5.3 - AsynchronousEvents

2011-12-14T08:47:43Z

Version 6.5.3 of The Server Framework was released today.

This release updates the WebSockets Option pack to the final version of the protocol as detailed in RFC 6455 which was released yesterday. There is also a bug fix to WebSocket status reason processing. If you have 6.5 or 6.5.1 or 6.5.2 and you are NOT using WebSockets then you probably don't need this release.

This release includes the following, see the release notes, here, for full details of all changes.

Updated to support RFC 6455 - added close status codes 1011 and 1015.
Fixed a bug in the handling of long status result messages, we now truncate them correctly.

RFC 6455: The WebSocket protocol - Rambling Comments

2011-12-14T08:46:47Z

I know I've said this before, but now it's really done...

The WebSocket protocol is now an official RFC. There are a small number of changes between RFC 6455 and the draft WebSocket protocol version 17; the only important ones being he addition of two new close status codes. The rest is just a case of tidying up the draft.

There will be a 6.5.3 release of The Server Framework to include these changes.

New client profile: Smart Moves Software Systems - Online gaming - AsynchronousEvents

2011-11-23T08:38:32Z

We have a new client profile available here for a new client who selected The Server Framework to help it expand its online gaming platform to incorporate a WebSockets interface.

A new release of WASP, now with SSL/TLS support - AsynchronousEvents

2011-11-22T10:10:47Z

We've just released a new version of WASP, our pluggable application server platform. This release is built with release 6.5.2 of The Server Framework and includes support for secure TCP connections using SSL/TLS via our SChannel Option pack.

Setting up a secure TCP endpoint with WASP is easy, simply add the Secure configuration option to the <EndPoint> node like this:

<?xml version="1.0" encoding="Windows-1252"?>
<Configuration>
  <WASP>
    <TCP>
      <Endpoints>
        <EndPoint
          Name="Echo Server"
          Port="5050"
          HandlerDLL="[CONFIG]\EchoServer.dll"
          Secure="true">
        </EndPoint>
      </Endpoints>
    </TCP>
  </WASP>
</Configuration>

This tells WASP to secure the endpoint using a default certificate called "Wasp" that is located in the "MY" certificate store. You can add a self signed test certificate using the standard Microsoft "make cert" utility, makecert.exe and a simple script which creates and installs the correct type of certificate can be downloaded from here.

if you do not want to use a certificate called "Wasp" in the "MY" certificate store then you can configure the certificate used by adding the StoreName, CertificateName and UseMachineStore config values.

<?xml version="1.0" encoding="Windows-1252"?>
<Configuration>
  <WASP>
    <TCP>
      <Endpoints>
        <EndPoint
          Name="Echo Server"
          Port="5050"
          HandlerDLL="[CONFIG]\EchoServer.dll"
          Secure="true"
          StoreName="OurSpecialStore"
          CertificateName="OurCertificate"
          UseMachineStore="true">
        </EndPoint>
      </Endpoints>
    </TCP>
  </WASP>
</Configuration>

Testing your new secure endpoint can be done using either our OpenSSL server test or our SChannel server test. These are example clients that ship with The Server Framework and that allow you to create thousands of concurrent connections and control how they send data to a server. This is an easy way to build a test system for your server as all of the complexity of managing and controlling the connections is done for you and you simply have to adjust the messages that are generated and how the response validation is done. The default message that is built is an network byte order integer length prefixed message and so this program can be used to stress test WASP with either of the first two example plugins that were discussed in the tutorial.

You can download the SChannelEchoServerTest program from here. See our for details of how to run this tool.

Latest release of The Server Framework: 6.5.2 - AsynchronousEvents

2011-11-21T08:25:00Z

Version 6.5.2 of The Server Framework was released today.

This release adds some new functionality to the WebSockets Option pack and fixes some bugs in code that is only currently used by the WebSockets Option pack. If you have 6.5 or 6.5.1 and you are not using WebSockets then you probably don't need this release.

This release includes the following, see the release notes, here, for full details of all changes.

Bug fixes to JetByteTools::IO::CBufferChain and JetByteTools::IO::CSortedBufferChain.
Added JetByteTools::IO::CLockableBufferProcessor::PutBackAndLock().
Adjusted how JETBYTE_DUMP_STREAM_SOCKET_READ_TO_DEBUG_LOG, etc. work, you can now produce less debug data (faster) using JETBYTE_TRACE_STREAM_SOCKET_READ_TO_DEBUG_LOG, etc.
Removed duplication in WebSocket header processing.
Breaking Change JetByteTools::WebSocket::IProtocolHandler::OnConnectionClosed() has been added and should be called when the underlying socket connection is closed. The example servers have been updated so please compare these to your own servers to see the changes required.
Added overloads of JetByteTools::WebSocket::IWebSocket::WriteText() and JetByteTools::WebSocket::IWebSocket::TryWriteText() which take BYTE * of pre-formatted UTF8 data.
JetByteTools::WebSocket::TWebSocketBase now holds a reference to the underlying stream socket until the connection closes, this is because we can't rely on their being a read pending to hold the stream socket open, even if the client thinks there is, as the websocket may have data buffered and so not issue a physical read.
Adjusted JetByteTools::WebSocket::CAutoDetectProtocolHandler so that it doesn't have to always issue a read when set to accept a connection. This allows for connections which may also carry non-websocket traffic.
Adjusted how the HyBi protocol handler responds to errors. We now issue the 'correct' websocket layer close notification with the correct close code (we pass the latest Autobahn tests (0.4.3) which include tests for appropriate close codes) and notify client code via two new callbacks on JetByteTools::WebSocket::HyBi::IWebSocketServer, JetByteTools::WebSocket::HyBi::IWebSocketServer::OnError() and JetByteTools::WebSocket::HyBi::IWebSocketServer::OnClosed().
Bug fix to JetByteTools::WebSocket::HyBi::CProtocolHandler which deals with a race condition during inbound data processing.
Bug fix to JetByteTools::WebSocket::HyBi::CProtocolHandler and JetByteTools::WebSocket::Hixie76::CProtocolHandler which deals with close handling, we now wait for the other side's close response correctly.
Removed unnecessary size limitations in the TCHAR * based socket write methods.

Dropping support for Visual Studio .Net 2002 and 2003 - AsynchronousEvents

2011-11-09T11:25:24Z

We are dropping support for Visual Studio .Net 2002 from release 6.6 of The Server Framework which is due early next year. We don't expect that this will cause anyone any problems as this compiler became unsupported by Microsoft in 2009, and most native C++ people seemed to skip this release entirely.

We are also considering dropping support for Visual Studio .Net 2003. This compiler is still supported by Microsoft until 2013 but we expect that most people interested in native C++ will have switched to Visual Studio 2005 or 2008 at the earliest opportunity.

Feedback is essential here. If you DO still need Visual Studio .Net 2003 support then we need to know, otherwise we may bring forward its retirement date...

New client profile: Takion Technologies - Equity trading platform - AsynchronousEvents

2011-11-04T09:11:03Z

We have a new client profile available here for a client that we've had since 2006 and who use The Server Framework in their equity trading systems.

Windows 8 Registered I/O Buffer Strategies - AsynchronousEvents

2011-10-31T07:30:00Z

One of the things that allows the Windows 8 Registered I/O Networking Extensions, RIO, to perform better than normal Winsock calls is the fact that the memory used for I/O operations is pre-registered with the API. This allows RIO to do all the necessary checks that the buffer memory is valid, etc. once, and then lock the buffer in memory until you de-register it. Compare this to normal Winsock networking where the memory needs to be checked and locked on each operation and already we have a whole load of work that simply isn't required for each I/O operation. As always, take a look at this video from Microsoft's BUILD conference for more in-depth details.

RIO buffers need to be registered before use

The recommended way to use RIORegisterBuffer() is to register large buffers and then use smaller slices from these buffers in your I/O calls, rather than registering each individual I/O buffer separately. This reduces the book-keeping costs as each registered buffer requires some memory to track its registration. It's also sensible to use page aligned memory for buffers that you register with RIORegisterBuffer() as the locking granularity of the operating system is page level so if you use a buffer that is not aligned on a page boundary you will lock the entire page that it occupies. This is especially important given that there's a limit to the number of I/O pages that can be locked at one time and I would imagine that buffers registered with RIORegisterBuffer() count against this limit.

Here we can see some common problems when you fail to take the locking granularity of the system into consideration when registering I/O buffers for RIO.

To avoid locking more memory than you need to always align your buffers by allocating with VirtualAlloc(), or VirtualAllocExNuma(). Note that these both work in terms of pages of memory though the allocation size is specified in bytes, also note the alignment restrictions; the start of each allocated block will be on a boundary determined by the operating system's allocation granularity (see here for why). You can obtain both page size and allocation granularity from a call to GetSystemInfo().

So, to allocate buffers for RIO efficiently you should use VirtualAlloc() to ensure alignment and you should allocate blocks which are multiples of the operating system's allocation granularity (or you'll be leaving unusable holes in your memory area). Some code like this might work:

SYSTEM_INFO systemInfo;

GetSystemInfo(&systemInfo);

const DWORD gran = systemInfo.dwAllocationGranularity;

const DWORD bufferSize = RoundUp(requestedSize, gran);

char *pBuffer = reinterpret_cast<char *>(VirtualAlloc(
   0,
   bufferSize,
   MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE));

if (pBuffer)
{
   RIO_BUFFERID id = rio.RIORegisterBuffer(pBuffer, bufferSize);

   if (id == RIO_INVALID_BUFFERID)
   {
      // handle error
   }
}

Buffer slices

Once you've registered your buffer you then access it using RIO_BUFs. The RIO_BUF is a handle to a "slice" of a RIO buffer and is very similar to the familiar WSABUF that's used in normal Winsock calls. It's simply a buffer id, a start offset and a length. You might decide that you'll allocate one block of memory for your I/O buffers and that would be 64K in size (since 64K is the allocation granularity on x64 and x86) and then perform I/O in terms of fixed sized buffers of 4096 bytes, or whatever. You could then have a series of RIO_BUFs, the first points to offset 0 and has a length of 4096, the second points to offset 4096, etc.

Alternatively you might write a memory allocator that sits over your RIO registered buffers and allocates portions on demand for whatever size you need, although personally I favour using fixed sized I/O buffers. Many server designs may have messages which are naturally limited in size and work fine with single fixed sized buffers for send and receive operations. When your data is larger than a single buffer you can chain buffers together. For sending you can then send the chain using scatter/gather I/O and for receiving you can simply fill a buffer, allocate another and chain the buffers together so that you can process complete messages. We've been using fixed sized buffers in The Server Framework for over 10 years and they work just fine in general purpose situations as long as you can select a buffer size that makes sense for most operations. The one potential reason to want to use dynamic buffers for RIO is that whilst the header's show that RIOSend() supports scatter/gather I/O the documentation says that it doesn't and the Microsoft BUILD video implies that the extra parameters are 'reserved'. Even so, with RIO's strict limits on the number of outstanding operations (and the number of buffers used with those operations) using our current "send a chain of buffers" style of design may not work so well with RIO. We'll see. For now I'll be sticking to fixed sized buffers as I don't believe that placing a custom memory allocator over the top of the RIO buffer will perform well enough; it may well be that we end up with an allocator that can allocate a variety of fixed sizes from different pools...

An augmented RIO_BUF

The RIO_BUF is too simple a structure to manage complex buffering requirements. Since we'll likely be using I/O completion port based completion handling in our RIO code we'll need a way to manage the lifetime of the buffer slices that we're using. For example, the slice is active from the moment we allocate it to put outbound data into it, until the write completion occurs, at which time it could be returned to the allocator, or, reused directly. I've found that reference counting works well for this, it also allows for flexible server designs as, when reading, you can easily manage the extended lifetime of a buffer that you need to place in a chain, or pass off to another thread for processing. You can see the OVERLAPPED based buffer interface that we use for IOCP servers in The Server Framework here, I imagine that the RIO buffer interface that I come up with will be similar, but somewhat simpler and without the OVERLAPPED. So, we now have a RIO buffer slice that's managed by a separately allocated 'buffer' object.

Buffer slice management

Since our buffer allocator will provide fixed sized buffers and work in terms of large blocks of memory for registering with RIO. It's probably also worth allocating a single large block of memory for all of the corresponding buffer objects that will manage the slices. We would then use placement new to allocate each of the objects in the single large block. We'd end up with two blocks of memory, like this:

The first would contain all of the buffer objects that manage the reference counts on the slices, hold the RIO_BUF structures and contain any book-keeping data we might need, the second be registered with RIO and locked in memory. The allocator would then allocate new blocks of memory as it needs them and, if the buffer objects also manage a reference count on the allocator's data blocks, it could de-register and release buffers when they're no longer in use (as long as it's pooling enough buffers for later use...).

One thing to bear in mind here is that my design here is naturally driven towards being flexible and general purpose. The reason for this is that's the way The Server Framework works; yes it's always possible to get better performance from code that's designed specifically for one single server, but with our framework you can get a working server up and running very quickly and, in most situations, you'll find that you don't need to tune the code much at all for your specific server. Also I find that it's easier to get started this way...

As an alternative to a buffer allocator which manages pools of RIO buffers we might, instead want to allocate buffers per connection. After all, RIO limits the number of operations that can be pending at one time on a connection, so we could, in some situations, know exactly how many buffers we might need an assign them to the connection when it's established. Whilst this may perform better in some situations, per socket buffer pooling is something that can be added on top of a pooling buffer allocator (and this is currently being investigated in The Server Framework anyway), so we'll look at it later if necessary.

So, that's the idea... More on this when I have some code.

Windows 8 Registered I/O and I/O Completion Ports - AsynchronousEvents

2011-10-25T17:56:24Z

In my last blog post I introduced the Windows 8 Registered I/O Networking Extensions, RIO. As I explained there are three ways to retrieve completions from RIO, polled, event driven and via an I/O Completion Port (IOCP). This makes RIO pretty flexible and allows it to be used in many different designs of servers. The polled scenario is likely aimed at very high performance UDP or High Frequency Trading style situations where you may be happy to burn CPU so as to process inbound datagrams as fast as possible. The event driven style may also help here, allowing you to wait efficiently rather than spin, but it's the IOCP style that currently interests me most at present as this promises to provide increased performance to more general purpose networking code.

Please bear in mind the caveats from my last blog post, this stuff is new, I'm still finding my way, the docs aren't in sync with the headers in the SDK and much of this is based on assumption and intuition.

How do RIO and IOCP work together?

RIO's completions arrive via a completion queue, which is fixed sized data structure that is shared between user space and kernel space (via locked memory?) and which does not require a kernel mode transition to dequeue from (see this BUILD video for more details on RIO's internals). As I showed last time, you specify how you want to retrieve completions when you create the queue, either providing an event to be signalled, an IOCP to be posted to or nothing if you will simply poll the queue. When using an IOCP you get a notification sent to you when the completion queue is no longer empty after you have indicated that you want to receive completions by calling RIONotify().

Simplified code for handling an IOCP driven RIO completion queue might look like this:

if (::GetQueuedCompletionStatus(
   hIOCP, 
   &numberOfBytes,
   &completionKey,
   &pOverlapped,
   INFINITE))
{
   const DWORD numResults = 10;

   RIORESULT results[numResults];

   ULONG numCompletions = rio.RIODequeueCompletion(
      queue,
      results,
      numResults);

   while (numCompletions)
   {
      for (ULONG i = 0; i < numCompletions; ++i)
      {
         // deal with request completion...
      }

      numCompletions = rio.RIODequeueCompletion(
         queue,
         results,
         numResults);
   }

   rio.RIONotify(queue);
}

Of course, in real code you'd likely use the completionKey to pass details of the queue that's being operated on, a pointer to a structure that you can plug the queue into once you've created it, perhaps.

Anyway, the gist of it is that once the RIO completion queue is not empty you will get an IOCP completion if you have called RIONotify() but you will not get another IOCP completion until you call RIONotify() again. At first this seems a little strange. After all, the completion queue could send another IOCP completion when it is, or becomes, non-empty once you have called RIODequeueCompletion() once. However, having to call RIONotify() to explicitly request a new notification is probably a good thing. It places you in complete control of which threads are currently accessing the RIO completion queue and given the nature of RIO completion queues this also means that by using the pattern above you can be sure that only one thread is processing completions for a given socket at a given time. Of course if you are using separate RIO completion queues for send and receive operations then you may have one IOCP thread processing send completions and one processing receive completions at the same time. If you use just one RIO completion queue for both send and receive then you can be sure that only this thread is currently processing completions for a given socket. This is different to the normal IOCP model where all of the threads in your I/O pool could be processing completions for the same socket if enough operations have completed.

Why is this a good thing?

This behaviour is good because it means that, hopefully, you need do nothing clever to retain sequencing in completions. Assuming reads and writes complete to the RIO completion queue in the expected order (and I'd be very surprised if they didn't) then the fact that you can guarantee that you only have one thread processing completions for a given socket means that you're guaranteed, at that point, at least, that the completions are in the correct order. With IOCP the completions are placed in the queue in order but the fact that one or more threads from your I/O pool could be processing completions for the same socket simultaneously means that you need to actively ensure that the completions are processed in order (if that matters to you, and, more often than not, it does). This is more important in RIO as (according to the RIO BUILD video) your socket's send and receive buffers are not used and so you need to have ample receives pending to ensure that you don't stall your TCP connection or lose UDP datagrams. With TCP, multiple pending receives require sequencing to ensure the stream is processed in the correct order.

It's also good because, quite frankly, there's no need to be notified again until you've drained the RIO completion queue and if completions keep arriving you get to stay on one thread and dequeue them. As I mentioned earlier, dequeuing completions doesn't involve a kernel mode transition and so we've suddenly switched to a polled design where we only need a kernel mode transition when we run out of work to do.

Why is this a bad thing?

Unfortunately this behaviour may make it a little harder to get full utilisation of all your I/O threads. You need to make sure that you have enough RIO completion queues so that each of your I/O threads can do some work and you need to hope that you've spread your connections across your queues in such a way that one RIO completion queue (i.e. a subset of connections) doesn't have more work to do than others. I expect the way that a general purpose framework assigns new connections to RIO completion queues and how many queues and of what size will all be things that I'll work out over time with some experimentation. If you have more RIO completion queues than you do I/O threads then you also need to be careful that you don't stick your thread to one completion queue by looping for long periods on a single RIO completion queue; I expect that a configurable limit on how many RIO results to process per IOCP completion would do the trick.

Something else to think about is the fact that now if an I/O thread blocks before it has called RIONotify() then you're blocking all of the sockets that are associated with the RIO completion queue that you're currently processing, not just a single connection.

Wrapping up

It looks pretty easy to scale RIO completion processing using IOCP notification but the details are not going to be the same as you're used to with normal IOCP completions. Each IOCP completion represents a potentially infinite block of RIO results for a subset of your connections. Expect your RIO architectures to be familiar, yet different to what you're doing now with IOCP.

Inside the Windows 8 Registered I/O Extensions, RIO - Rambling Comments

2011-10-24T20:30:38Z

Before I started to look at RIO for inclusion in The Server Framework I did a quick check on the Microsoft BUILD site to see if there were any sessions that dealt with it specifically, I didn't find any. Once I posted my blog posting I did another check and found this video that deals specifically with RIO. This gives some in depth details of how RIO works and the kinds of performance improvements that Microsoft has witnessed in their labs. It's interesting and impressive.

One thing that I hadn't realised is that with RIO, SO_SNDBUF and SO_RCVBUF are no longer applicable with RIO, it's as if you're operating with them set to zero. You need to always have recvs pending if you don't want to drop datagrams...

More once I've watched it all...

Windows 8 Registered I/O Networking Extensions - AsynchronousEvents

2011-10-24T15:50:35Z

Most of the buzz being generated around the Windows 8 Developer Previews at the moment seems to be centred on the new Metro user interface and the Windows Runtime. Whilst both Metro and WinRT are key components of the next Windows release I find the Registered I/O Networking Extensions to be far more interesting, but then I guess I would...

What are the Registered I/O Networking Extensions?

The Registered I/O Networking Extensions, RIO, is a new API that has been added to Winsock to support high-speed networking for increased networking performance with lower latency and jitter. These extensions are targeted primarily for server applications and use pre-registered data buffers and completion queues to increase performance. I assume that the increased performance comes from avoiding the need to lock memory pages and copy OVERLAPPED structures into kernel space when individual requests are issued, instead relying on pre-locked buffers, fixed sized completion queues, optional event notification on completions and the ability to return multiple completions from kernel space to user space in one go.

The RIO API is pretty simple and straight forward but servers that currently use I/O Completion Port based designs will need to change somewhat to take advantage of it and probably not all server designs will benefit from changing. RIO relies on you registering the memory that you will use as data buffers and knowing in advance how many pending operations a given socket will have at any time. This allows it to lock the data buffers in memory once, rather than on each operation and removes the whole concept of the OVERLAPPED structure from the user space API. Since completion queue space is also of a fixed size you're also required to know how many sockets you will be allocating to a given queue and the maximum number of pending operations that these sockets will have. You can increase all of these limits after socket creation but, except for registering new data buffers, I expect that you're likely to take a performance hit for doing so.

I've been looking at the pre-release documentation and the headers from the latest Windows SDK and experimenting with the new RIO API. Note that at present the documentation is out of sync with the headers and there's little more than reference documentation so much of what I have to say about RIO is based on assumptions and intuition based on the available information and my knowledge of how I/O Completion Port based networking currently works on pre Windows 8 operating systems. In other words, don't rely on all of this to be correct.

How do you access the RIO API?

RIO is a Microsoft-specific extension to the Windows Sockets specification in the same way that AcceptEx(), ConnectEx(), etc. are and the API is accessed in a similar way. You don't link to the functions directly, you obtain them via a call to WSAIoctl(). Since RIO presents an API rather than a single extension function and that API is either available as a whole or not you obtain a table to the API's function pointers, rather than individual function pointers as with AcceptEx() and ConnectEx() etc. You do this by calling WSAIoctl() with an opcode of SIO_GET_MULTIPLE_EXTENSION_FUNCTION_POINTER and an id of WSAID_MULTIPLE_RIO. The result is a populated RIO_EXTENSION_FUNCTION_TABLE table, see here for more details.

It took me a few attempts to get the WSAIoctl() call to work as this is the first extension API and the first use of SIO_GET_MULTIPLE_EXTENSION_FUNCTION_POINTER and I was unable to find any examples of its usage. Anyway, your call to WSAIoctl() should look like this:

RIO_EXTENSION_FUNCTION_TABLE rio;

GUID functionTableId = WSAID_MULTIPLE_RIO;

DWORD dwBytes = 0;

bool ok = true;
 
if (0 != WSAIoctl(
   s,
   SIO_GET_MULTIPLE_EXTENSION_FUNCTION_POINTER,
   &functionTableId,
   sizeof(GUID),
   (void**)&rio,
   sizeof(rio),
   &dwBytes,
   0,
   0))
{
   const DWORD lastError = ::GetLastError();

   // handle error...
}
else
{
   // all ok, we have access to RIO
}

Note that you can use the cbSize member of the function table to detect additions to the API if it is changed in later versions of Windows.

First impressions of the RIO API

Looking at the preliminary on-line documentation a couple of things immediately jumped out at me:

None of RIOReceive(), RIOReceiveEx(), RIOSend() and RIOSendEx() support scatter/gather I/O. That is, they all take a single RIO_BUF structure rather than a chain of them. Standard Winsock send and recv functions take chains of WSABUF structures allowing for scatter/gather I/O.
Completions are signalled via an event. Each completion queue can have its own event associated with it and the pattern for retrieving completions appears to be, issue operations, wait on event, retrieve completions. Whilst this is most likely the most performant method for small numbers of connections it leaves you having to scale it yourself which is likely non-trivial.

Luckily the header files are not consistent with the documentation and the fact that they DO include support for scatter/gather I/O and they also allow completion notification either via an event or via IOCP means that I'm pretty sure that the code is right and the docs are wrong. Anyway, I'm getting ahead of myself...

How does RIO work?

As I mentioned above, RIO provides increased performance by working with pre-locked buffers, fixed sized completion queues and reduced user mode to kernel mode transitions. You enable the RIO extensions on a socket by creating the socket with the WSA_FLAG_REGISTERED_IO flag, it seems that this can be combined with WSA_FLAG_OVERLAPPED which is convenient as RIO provides no alternatives to AcceptEx() and ConnectEx() and so it's likely that your sockets will require both WSA_FLAG_REGISTERED_IO AND WSA_FLAG_OVERLAPPED.

SOCKET s = ::WSASocket(
   AF_INET,
   SOCK_STREAM,
   IPPROTO_TCP,
   NULL,
   0,
   WSA_FLAG_REGISTERED_IO);

Once you have your socket you need to create a registered I/O socket descriptor on the socket. You do this by calling RIOCreateRequestQueue(). The documentation for this function is out of sync with the headers, the call actually looks like this:

ULONG maxOutstandingReceive = 10;
ULONG maxReceiveDataBuffers = 1;
ULONG maxOutstandingSend = 10;
ULONG maxSendDataBuffers = 2;

void *pContext = 0;

RIO_RQ requestQueue = rio.RIOCreateRequestQueue(
   s,
   maxOutstandingReceive,
   maxReceiveDataBuffers,
   maxOutstandingSend,
   maxSendDataBuffers,
   recvQueue,
   sendQueue,
   pContext);

This is where you place limits on the number of outstanding requests (and the number of buffers that can be used with each request) and where you associate your per-connection context that will be returned to you with each completion; this is the same as the "completion key" in regular IOCP designs. You also need a receive queue and a send queue (you can use one queue for both), these queues are created by a call to RIOCreateCompletionQueue(). Again the documentation is out of sync, the call looks like this:

HANDLE hEvent = WSACreateEvent();

RIO_NOTIFICATION_COMPLETION type;

type.Type = RIO_EVENT_COMPLETION;
type.Event.EventHandle = hEvent;
type.Event.NotifyReset = TRUE;

RIO_CQ queue = rio.RIOCreateCompletionQueue(queueSize, &type);

Which creates a completion queue of a specified size which uses an event to signal that it's no longer empty. When you create a request queue the number of outstanding operations is used to ensure that the queue size is suitable for all the sockets associated with it. You can resize both completion queues and request queues at a later time if you need to but I would imagine that it's better not to base your design on doing so.

As an alternative you can use an IOCP for completion notification.

HANDLE hIOCP = CreateIoCompletionPort(
   INVALID_HANDLE_VALUE,
   0,
   0,
   0);

OVERLAPPED overlapped;

RIO_NOTIFICATION_COMPLETION type;

type.Type = RIO_IOCP_COMPLETION;
type.Iocp.IocpHandle = hIOCP;
type.Iocp.CompletionKey = pCompletionKey;
type.Iocp.Overlapped = &overlapped;

RIO_CQ queue = rio.RIOCreateCompletionQueue(queueSize, &type);

This makes it easier to scale the use of RIO with a pool of IOCP threads processing completions from one or more queues. Of course the overlapped structure should likely be dynamically allocated so that it can last until the queue is closed.

The header file parameter annotations suggest that the completionType parameter is optional, thus there's a third way to create a completion queue.

RIO_CQ queue = rio.RIOCreateCompletionQueue(queueSize, 0);

Which seems to provide a polled interface...

Receiving data using RIO

Once the socket is connected using normal connection methods you can send and receive using RIO. The two receive functions available in the Windows Developer Preview SDK differ from the documentation in that they DO support scatter/gather I/O. The simplest recv call looks like this:

RIO_BUF buffer;

buffer.BufferId = id;
buffer.Offset = 0;
buffer.Length = buffer1Size;

DWORD flags = 0;

void *pOperationContext = 0;

rio.RIOReceive(requestQueue,
   &buffer,
   1,
   flags,
   pOperationContext);

The RIO_BUF structure allows us to create a buffer slice from a registered data buffer. This lets us register large buffers, which is more efficient, and lets us slice them up into blocks that are more appropriate to use. A buffer is registered like this:

const DWORD bufferSize = 4096;

char *pBuffer = new char[bufferSize];

RIO_BUFFERID id = rio.RIORegisterBuffer(pBuffer, bufferSize);

Note that, of course, the buffer should outlive the buffer registration and that it would be better to allocate memory that is page aligned, using VirtualAlloc() as buffer registration locks the buffer in memory and the locking granularity is page sized. See the documentation for RIORegisterBuffer() for more details.

The operation context is your per-operation data, this is what you would have previously used your 'extended' OVERLAPPED structure for. In a real design this is likely a pointer to a reference counted 'operation data' object which knows about the buffer slices being used and the operation, in this case a read, that is being executed.

So, what happens when RIOReceive() completes? Well, if we're dealing with the event based completion mechanism, then, at present, you need to call RIONotify() to tell the RIO API that you want to receive notifications when completions occur (once again the docs for this function are out of sync with the implementation). You then wait on your event until it's signalled and then call RIORIODequeueCompletion() to retrieve completion results. Like GetQueuedCompletionStatusEx() you can remove multiple completions in a single call, in this case, by passing an array of RIORESULT structures to this call.

RIORESULT result;

ULONG numResults = rio.RIODequeueCompletion(queue, &result, 1);

Here we request a single completion, but for better performance it's probably best to always work with arrays of completions. We can now wait for another completion, but, at least at present, we need to call RIONotify() again to request more notifications, we can't simply reset our event and wait again. It seems strange that we have to call RIONotify() manually when RIO_NOTIFICATION_COMPLETION has a boolean member called NotifyReset, but, at present we do. I would expect that by setting NotifyReset to true the act of dequeuing completions would reset the event AND enable further notifications, thus avoiding another potential kernel mode transition.

There are some flags that you can specify in your RIOReceive() call. The most interesting is RIO_MSG_WAITALL which causes the recv call to only complete when the buffer slice supplied is full, an error occurs, or the connection is terminated. This would be very useful for servers that work with messages which are of a fixed length, or are framed with a length prefix. By supplying a buffer of the appropriate size and specifying RIO_MSG_WAITALL you'll get a single completion when the buffer is full. This is considerably better than getting a completion with a partial buffer and then needing to adjust the start position of the buffer and reissue the read so that you can read the rest of the message into the buffer. The reduced number of completions that need to be processed in this scenario, especially with large messages, would likely turn into a huge performance gain.

Note that currently, in a multi-buffer slice read operation, RIO_MSG_WAITALL will cause a completion to occur when the first buffer slice is full, not when all buffer slices supplied in the call are full.

Sending data using RIO

Sending data is pretty much the same as receiving it, you pass an array of buffer slices to RIOSend() and process the completion in the normal way.

RIO_BUF sendBuffer;

sendBuffer.BufferId = id;
sendBuffer.Offset = 0;
sendBuffer.Length = bufferSize;

// memcpy your data into the buffer slice...

if (!rio.RIOSend(requestQueue, &sendBuffer, 1, flags, pContext))
{
   // handle error
}

Conclusions

I've only scratched the surface of RIO here and the fact that the documentation is out of sync with the actual implementation means that this could all change before Windows 8 is released, but... Although RIO will likely mean that your design is more complicated than a "normal" IOCP design, I expect the performance gains will be worth it for certain types of networking applications. Being able to pre-lock all your memory buffers in memory and pre-assign your completion queues likely means that your server will be more robust, with failures due to the I/O page lock limit and non-paged pool exhaustion becoming a thing of the past. Processing completions on dedicated threads using the eventing version of the API is likely to result in higher performance for applications that suit that design whilst using the IOCP based notification system will scale more easily. I can see two use cases for RIO and I'm sure there are many more. The first is for high performance, jitter free, low latency connections where you have a small number of connections to deal with an want the best performance possible. The second is for servers with many thousands of concurrent connections where the performance gains from streamlined send and recv APIs and the correspondingly reduced kernel transitions lead to higher throughput on all connections.

We certainly intend to incorporate support for RIO into The Server Framework and right now we're investigating the best way to do this. I'll be blogging more about RIO over the next few weeks, why not subscribe to our RSS feed or follow us on Twitter.

Out of band data, TCP Urgent mode and overlapped I/O - AsynchronousEvents

2011-10-20T09:30:00Z

Some stream protocols have the concept of 'out of band' (OOB) data. This is a separate logical communication channel between the peers which enables data that is unrelated to the current data in the stream to be sent alongside the normal data stream. This is often a way for some data to jump ahead of the normal stream and arrive faster than if it were delivered via the the normal data stream.

Winsock supports out of band data in a protocol independent way, see here, but accessing it from networking code that uses overlapped I/O rather than the old-fashioned BSD API is somewhat under documented. By default, out of band data does not appear in the normal data stream, you have to read it explicitly by setting MSG_OOB in the flags of a call to WSARecv(). For non overlapped I/O designs you can use WSAAsyncSelect() or select() to explicitly check for the presence of out of band data. With overlapped I/O your options appear limited, it seems that you should be able to use an overlapped WSAIoctl() call with SIOCATMARK but this will return immediately when OOB is either present or not present, it doesn't wait for OOB to become available.

The solution is to post a separate, out of band, overlapped WSARecv() passing the MSG_OOB flag. This will only return on socket closure or when out of band data arrives. By using a distinct indicator in your 'per operation data' you can distinguish this read from normal reads and deal with it accordingly. Once you have processed the special out of band data read you can then post another to read subsequent out of band data.

Inline Out Of Band Data

Winsock also allows you to include any OOB in the normal data stream, you can do this by setting the SO_OOBINLINE socket option on the connection. When SO_OOBINLINE is set the OOB data is included in the results of normal WSARecv() calls without needing to set MSG_OOB in the flags, however, with overlapped I/O it's then impossible to determine if OOB exists and if it's present in the data buffer returned by any particular WASRecv() call. When SO_OOBINLINE is set calls to WSAIoctl() with SIOCATMARK always return true to indicate that no OOB is waiting. For most Winsock Providers this is probably fine. You most probably want to read your OOB separately for X.25 or whatever and the last thing you want is for it to be mixed in with the normal data from the data stream.

For TCP things are somewhat more complex. TCP doesn't actually have the concept of OOB, but the BSD API maps the TCP Urgent data concept onto OOB. Due to differences in the TCP RFCs and historical issues there's complexity in using TCP Urgent data reliably if you don't know which RFC the implementation adheres to. This complexity makes it almost impossible to use OOB with overlapped TCP in a reliable manner, especially if you have no control over the platform upon which your clients run.

TCP Urgent Data

In TCP out of band data is implemented in terms of 'urgent data' using the URG bit and the Urgent Pointer, see here, however, there are two conflicting descriptions of how this works, RFC 793 which details TCP says that the Urgent Pointer indicates the byte that follows the urgent data but RFC 1122 corrects this and states that the Urgent Pointer indicates the final byte of urgent data. This leads to interoperability issues if one peer uses the RFC 793 definition and the other uses the RFC 1122 definition. The Windows documentation for the standard TCP Winsock Provider claims that it operates as BSD does however this can be changed using the TCP_EXPEDITED_1122 socket option if the Winsock provider supports it. There's more compatibility complexity in that Windows only supports a single byte of out of band data whereas RFC 1122 specifies that TCP MUST support sequences of urgent data bytes of any length. Windows also doesn't specify how or if it will buffer subsequent out of band data, so if you are slow in reading a byte of urgent data and another byte of urgent data arrives then one of the bytes may be lost; though our tests have shown that Windows does buffer urgent data. This all makes the use of out of band signalling using urgent data somewhat unreliable on Windows with TCP.

The unreliability shows itself if you're using separate OOB reads using MSG_OOB. In this instance you may get the 'wrong' byte of OOB data if you are dealing with a peer that implements the altenative RFC. This wouldn't be a problem if the byte hadn't been removed from the normal data stream and so you end up with an incorrect byte in the normal data stream (the intended OOB byte) and a missing byte, which has been treated as OOB data. This doesn't cause problems for the two traditional users of TCP Urgent Data, Telnet and Rlogin as they simply use the Urgent data notification to alert the server that urgent data is present in the data stream and then read and discard all of the normal data in the stream until they can read the urgent data. See here for how this facility is used in the Telnet protocol's synch function. It's practically impossible to implement this functionality with overlapped I/O as you don't want the urgent data removed from the data stream, so you need to operate in SO_OOBINLINE mode and yet when in that mode you will never be notified that urgent data exists. To implement the telnet model with overlapped I/O you pretty much need to always read all inbound data and buffer it in your server rather than allowing it to buffer in the TCP stack and allowing TCP flow control to prevent the client from sending more until you're ready.

So, if you want to use out of band data with TCP with overlapped I/O you need to remember five things:

Out of band data in TCP on Windows when interoperating with other, non-Windows, operating systems is likely to be unreliable due to differences between RFC 793 and RFC 1122.
Expecting to send more than a single byte of out of bound data is likely not to work.
Your out of band data may get "lost" if you send out of band data faster than the receiver is processing it.
To read out of band data using overlapped I/O you need to post a special WSARecv() with MSG_OOB set in the flags.
Real out of band communication using TCP is better achieved with a separate 'control' connection rather than using TCP's Urgent data function via MSG_OOB.

Despite all of this, The Server Framework will support out of band data in the next major release. By default OOB data will be disabled and we'll abort connections that use it. You can also decide to enable OOB inline which effectively disables OOB data from the receiver's point of view and simply keeps the out of band data in the main data stream. Or, you can enable async OOB which will read OOB data using an optional special purpose overlapped read with MSG_OOB set.

Latest release of The Server Framework: 6.5.1 - AsynchronousEvents

2011-10-11T07:23:31Z

Version 6.5.1 of The Server Framework was released today.

This is primarily a bug fix release, although we also add several new example clients and servers.

This release includes the following, see the release notes, here, for full details of all changes.

Bug fixes to The Core Framework which affect the use of the newly added "Read Again" functionality.
Fixes to the Hixie76 WebSockets protocol handler to improve interoperability.
Added outbound connection establishment support to the Hixie76 protocol handler.
Updated the WebSocket Echo Server Test and the Secure WebSocket Echo Server Test example clients to support the creation of both Hixie and HyBi connections.
Fixed a race condition in the WebSocket example clients that could cause a connection to "stall" - note that this was an issue with the how the client code used the WebSocket layer and not an issue in The WebSockets Option Pack itself.
Added two new example servers; a secure, managed WebSocket server which hosts the CLR and routes complete WebSocket messages to managed code for processing and a version of this example which also hosts a simple HTTP server.
Added a new HTTP client. This is designed to stress test HTTP and HTTPS servers by creating thousands of concurrent connections and requesting various resources at controllable rates.