Rambling Comments

Major Vista Overlapped I/O change

2008-02-18T08:17:29Z

I'm still reading the Richter book, highly recommended even if you've read one of the earlier versions. In fact it's possibly MORE highly recommended IF you've read one of the earlier versions... It seems that lots of things have changed a little, and some things have changed a lot. Unfortunately the book doesn't detail the actual changes. Note to publishers; I'd PAY for a slim book that DOES detail the changes between the APIs that are being discussed...

Take this throwaway line in the Cancelling Queued Device I/O Requests section of the Asynchronous Device I/O chapter of the latest book: "When a thread dies, the system automatically cancels all I/O requests issued by the thread, except for requests made to handles that have been associated with an I/O completion port." This is then clarified later in the chapter in a note which points out that prior to Windows Vista if you associated a device with an I/O completion port and then issued overlapped I/O requests on it then you had to make sure that the thread that issued the requests remained alive until the I/O requests had completed. Not anymore! Vista now allows threads to issue overlapped I/O requests and exit and it will still process the requests and queue them to the completion port. This makes perfect sense and will simplify writing general purpose I/O completion port code.

When I designed The Server Framework I decided that I couldn't require users of the framework to use my own brand of thread start and thread termination functions so that I could keep track of device I/O requests that their threads may have issued. What's more, the first server that I designed with the framework had a flexible thread pool for database access and could easily create a thread, issue an I/O request and then shut the thread down before the request completed. To avoid these issues I added code that "marshalled" all I/O requests into the I/O thread pool (the pool of threads that serviced the I/O completion port, which was fixed in size and existed for as long as the I/O completion port existed). Thus I/O requests were passed across to the I/O threads and issued from there to avoid the thread termination causes cancelled I/O issue. It seems that I can improve performance again by removing this indirection if we're running on Vista or above...

However, it seems that the MSDN documentation hasn't caught up yet, see the documentation for WSASend(), http://msdn2.microsoft.com/en-us/library/ms742203.aspx which says this in a note in the Overlapped Socket I/O section; "Note All I/O initiated by a given thread is canceled when that thread exits. For overlapped sockets, pending asynchronous operations can fail if the thread is closed before the operations complete. For more information, see ExitThread."

If someone locates some information on an MSDN page that confirms Richter's position then please let me know. In the meantime I'll put some changes into the 5.3 release of The Server Framework and start running some tests. This should result in some quite significant performance improvements from some server designs.

Currently Reading: Windows via C/C++

2008-02-11T08:33:46Z

I've just picked up a copy of Windows Via C/C++ (PRO-Developer) by Jeffrey Richter and Christophe Nasarre. This is 'version 5' of the book that started out as Advanced Windows NT (Advanced Windows). The book has been updated for Windows Vista and other changes that have happened since the last version, Programming Applications for Windows (Microsoft Programming Series).

I've decided to read it from cover to cover to refresh my knowledge and pick up on any changes. It's nice to see that even in the first section on error handling the text has been checked and updated, for example, the common FormatMessage() usage problem that Raymond Chen mentioned a while back (see here) is included; but interestingly it seems that the previous version of the book didn't specify FORMAT_MESSAGE_IGNORE_INSERTS flag and so, was probably the root cause of the bug in my code in the first place ;).

Currently reading: Excel add-in development in C/C++

2008-01-09T09:57:37Z

As I mentioned a while back, I'm writing a managed XLL style add-in system for Excel for one of my clients at the moment. This is going pretty well, most of the custom marshalling code is now done and we can write code in C# and expose it to Excel as worksheet functions.

Over Christmas I picked up a copy of Excel Add-in Development in C/C++: Applications in Finance by Steve Dalton. This is a book that I wished I had back in 2001 when I first started writing XLLs. It's full of otherwise hard to find information about how you work with, and around, the C API that Excel exposes for the development of XLL add-ins. So far, every time I remember a gotcha from my previous XLL development projects I can look up a solution for it in the book. I'm also learning a lot of useful things about how I can stretch the C API even further than I did on my last projects. The books is pretty dense; I'm not sure how approachable it would be to someone who'd never developed an XLL, though it does come with a CD of sample code and it explains XLLs from first principles (including how to set up your first XLL project in various versions of Visual Studio). It's a good reference book though (and that's what I need).

Highly recommended.

Charles Petzold on why he loves books

2006-01-19T08:01:42Z

Charles Petzold confesses his love for books.

I must admit this bit really hit home for me: "I love how my books remind me of passages in my life. I love the shelves of authors I've been obsessed with, and the books that knocked me over. I love knowing that I still own virtually every book I've read."

I have shelves and shelves of technical books as I've always been quite happy to buy a book on the chance that I might learn one thing from it. I have some that I didn't really learn anything from but they're few and far between. It's amazing that I can often get up and reach straight for the book that holds the answer to my current problems, and it's equally amazing the memories that doing so often conjours from the previous times that I've lifted the volume from the shelf...

I dropped out of the habit of reading fiction a few years ago and spent most of my time reading technical stuff, I'm now slowly getting back into fiction. Because of the gap, most of my old fiction is boxed up and stored away. It's strangely reassuring to know that I still have all the stuff that I was reading back when I was a teenager. I may never want to read it again, but if I do, it's there: The huge collection of Harry Harrison. The piles of fantasy fiction. The beaten up copy of "Star Wars" (with its bizarre section of colour stills from the film in the center pages), and its stained cover from Christmas day when I was 10 or 12 ...

Somehow all the electronic text, CD Roms, pdfs, etc, that I have accumulated over the years doesn't quite have the same attachments...

Still, the Sony Reader is tempting in its own way... It would be nice to be able to have each book I buy come with an electronic version that would work on something like the Sony Reader.

Book review: Rootkits by Hoglund and Butler

2005-09-28T07:27:47Z

I've just finished reading Rootkits: Subverting the Windows Kernel. Overall I enjoyed it but I'm in two minds about its usefulness...

The book covers "hacking" the Windows kernel using various techniques to get your code inside the kernel and, once there, various other techniques to keep others from knowing you're there. Once you have this kind of code in the kernel of someone's machine you can do lots of damage; or, of course, you can gather information for security forces, or protect the machine from other rootkits...

The book doesn't come with source code but instead refers to real rootkits that are available for download on www.rootkit.com. The subject matter is very interested and quite a few times I found myself thinking "wow, if the virus writers start doing this on a large scale then we're in big trouble". I guess that's the point of the book. It can be read at many levels and there's enough information in there to scare a manager into taking security more seriously... I liked the way that the code and concepts were presented. It's not a "for dummies" book, it doesn't hold your hand, it skims through the "tricks" and techniques at quite a fast pace and if you wanted to write your own rootkit then you'd have to put in a lot more work to do so; which is good... My virus checker instantly spotted the exe signatures of the rootkits that I built from the source on the web site which makes them safer for use as study aids... Overall the book feels a lot like the old style "cook book" coding books that I used to read when I was first learning C; there are lots of little techniques that you can take further. This book just whets your appetite.

Chapter one explains the hows and whys of rootkits. If you have code in the kernel you can manipulate what the kernel reports on its internal status and thereby hide your presence. Don't want your process to show up on the task list? Just monitor the calls that taskmon uses and remove your process (and remember to add the process time to some other process..) Don't want your log file to be discovered? Just subvert the filesystem and hide that file. Etc.

Chapter two builds a simple driver and shows you how to load it into the kernel. This is quite a good intro into building a driver and getting the environment set up. Once we can load a driver the next thing to do is hide the driver inside the loader executable as a resource, the idea being that one file is easier than two...

Chapter three presents a quick intro into the hardware. It covers the x86 access control concepts, memory pages, address translation, descriptor tables, call gates and interrupts. This is background reading for the next chapter...

Chapter four covers hooking API calls. It starts with hooking in user mode, something that is well documented, and moves on to hooking in kernel mode, something that is far less well documented. The details on user mode hooking can only really be considered as an overview, if you're interested in this stuff you should continue with Programming Applications for Windows by Jeffrey Richter. Next we move on to kernel mode hooking. This is the kind of thing that the SysInternals Regmon and Filemon tools use to do their work. There's more detail here, and sample code and a link to a downloadable example that hooks the kernel and hides processes from task manager. Next we're hooking interrupts, and finally we're hooking other drivers; again there's more sample code.

Chapter five covers runtime patching. We're now into the realm of direct binary modification of the code that runs. This is the kind of stuff that crackers may use to avoid your copy protection (I expect I just hosed this articles Google ads with that ;) ). The chapter covers how to locate the function call that you want to patch when it's not exported by looking for unique bytes in the function signature and how to safely patch in a detour to your own code. Once again there are source examples to download.

Chapter six deals with layered drivers; the Windows driver model is very flexible, allowing drivers stacks to be built by layering one driver on top of another. The stack processes the request and each layer can modify the request, or response, if it wishes to. This chapter features a detailed explanation of how to craft a keyboard sniffer. This is a non trivial project and the code and commentary includes lots of hints and tips that would be useful for writing more legitimate drivers. Next we're on to file filter drivers and it's at this point that I feel the book loses its edge. There's no code to download for the file filter driver and the discussion is more theoretical than the keylogger commentary. There's more code in the text but since you can't grab something that runs it's less use as a learning aid, IMHO.

Chapter seven talks about direct kernel object manipulation. API hooks are relatively easy for anti-virus and rootkit detection programs to find. If you don't want to be found then it's "better" to modify kernel data structures directly... There's an example of hiding a process by removing it from the kernel's list of processes. Examples of increasing a processes' access rights and adding SIDs to a process and examples of faking the Windows event viewer. This chapter was less interesting to me as I'm not actually interested in using this knowledge for devious reasons and this stuff had less legitimate uses, IMHO.

Chapter eight covers hardware manipulation. We're into the realms of modifying the firmware on a network card, or hacking the BIOS. Most of the coverage is from a very high level, though we do drop down to some code examples for accessin the keyboard controller directly. Again I wasn't especially interested in the subject matter.

Chapter nine deals with networking and provides some examples of accessing the network from within your driver. Starting with TDI we're shown how to use the TCP/IP stack from within a driver. Next we're on to raw sockets and finally we look at writing our own NDIS protocol driver.

Chapter ten is a, relatively slim, chapter on detecting rootkits. It suggests how to find out if there's something hiding in your kernel by looking for hooks and modifications etc. I think, perhaps, this would have been better if it had been presented alongside the 'how to hack' information, rather than as an afterthought at the end.

The book's a good read. If you are interested in doing bad stuff then you've probably already found much more information on the web, but this stuff is presented in a "cook book", tutorial style, and you can read it in the bath ;). The book is useful even if you have no interest in writing code to subvert the Windows kernel. It has lots of interesting stuff in there and, if you're interested in writing drivers, it's a bit easier to get into than the "normal" driver writing books; it's less formal and gives a whirlwind tour through what's possible rather than providing a solid description of how to do it properly. Due to the fact that the goal of the book is not to teach you how to write a device driver it gets to touch on lots of interesting snippets. From a security perspective the book is certainly an eye opener. If you're in the business of computer security you have to understand this stuff to stand a chance at being able to protect yourself and your clients from it.

If you have a Windows system, you'll probably want to run this now...

Rootkits

2005-09-15T07:29:20Z

Just saw a mini review of Rootkits: Subverting the Windows Kernel over on Ted Neward's blog. Looks like the perfect follow up to Undocumented Windows 2000 Secrets as I continue my slow progress towards being able to write Windows device drivers... From the sound of Ted's review it's written in a similar way to the Undocumented secrets book; showing you how to write drivers which aren't for real hardware... Anyway, I popped over to Amazon and it was being recommended on the front page for me due to my past purchases. It's now ordered, more once it arrives.

Undocumented Windows 2000 Secrets

2005-08-24T07:57:01Z

Just finished reading Undocumented Windows 2000 Secrets: A Programmer's Cookbook by Sven B. Schreiber. Well, I say reading, it was really just a first pass through the book. The text and code spends more time in kernel mode than user mode; there's a lot of information in there and it's all relatively new to me so it's the kind of book that needs a fair bit of work.

Sven starts by explaining how to set up a system that you can use for kenel mode development, you don't want to use your main box as it will be blue screening a lot of the time. Thanks to virtual machines I had something set up pretty quickly. VMWare/Virtual PC are ideal for this as you can map the virtual machine's serial ports to named pipes on the host machine and windbg can (sometimes) connect to these so that you can run the debugger on your main machine... Once you have a machine set up the first thing you do is force it to blue screen using a simple, bad, driver. The book then walks you through capturing the crash dump and working with it with i386kd.exe and symbol files.

Next comes an explaination of the Windows 2000 Native API, what it is, how it works, how Win32 is just a layer on top and how you can interface parts of the Native API from Win32 code.

Next we're into the DDK and developing a simple skeleton device driver. This seems like a much better introduction to device driver development than either Programming the Windows Driver Model or Developing Windows NT Device Drivers: A Programmer's Handbook both of which I hope to return to now that I have a bit more knowledge under my belt.

The meat of the book is Sven's "spy driver" which takes up the rest of the book. The spy driver allows you to look at kernel memory and decode it, hook kernel APIs calls (like regmon and filemon do) and call arbitrary kernel mode functions from user mode. It's complex code but it's described really well. I didn't play with the spy driver that much, but I will when I read the book again... This time through it was enough to read about how it works and use it as an simple introduction to writing drivers.

Sven makes a complicated subject relatively approachable. The book isn't an easy read but the writing is good and everything is explained well. Recommended; pity it's out of print...

Currently reading

2005-07-01T09:59:42Z

Just before I dive back into my other project, the one I don't talk about, I thought I'd post a short note about the pile of books that I'm currently reading...

Working Effectively with Legacy Code by Michael C. Feathers. Roy Osherove has this as his book of the month and I'd previously read some of Michael's writings on testing and found them useful. The book's great. If you ever have to work with the kind of code that I wrote about at length as 'the refactoring project' then this book is for you. Michael presents methods for getting horrible legacy code into a test harness and does it in such a way that you have no argument for not following his advice. Chapters such as "I Don't Have Much Time and I Have to Change It", "I Can't Get This Class into a Test Harness" and "I Need to Make Many Changes in One Area", show you how to get some tests into the the code and start making things better. I'd recommend this to anyone who's working on legacy code, and as far as Michael's concerned any code without tests is legacy code.

Mastering Regular Expressions by Jeffrey E. D. Friedl is a book that I really should have read a long time ago. I've managed to avoid regular expressions in the past and I recently realised that I should stop avoiding and start understanding - thanks to Filip for guilting me into that when he laughed at my pathetic efforts at manual search and replace in the VC IDE a while back. I haven't got very far with the book yet but already the introduction has confirmed that I'm right in needing to know more about what should be a powerful tool in my toolbox.

Undocumented Windows 2000 Secrets: A Programmer's Cookbook by Sven B. Schreiber. A while back I was looking for some information on CodeProject and this book came up as a reference. I did some more searches and people were raving about the book but unfortunately it is out of print and hard to come by. I found a copy on Amazon.com's marketplace for a slightly scary price (a lot less scary than the price over on Amazon.co.uk now) and decided to get it anyway. It only arrived last night but from a quick skim through it looks like it will be a good one...

C++ Template Metaprogramming: Concepts, Tools, and Techniques from Boost and Beyond (C++ in Depth S.) by David Abrahams and Aleksey Gurtovoy. As I've said before, I'm not currently clever enough to use this complicated template metaprogramming stuff. I'm reading the book to become clever enough.

And finally...

Code Noir by Marianne De Pierres. Cyberpunk in future Australia. Guns, girls, motor bikes and helicopters, who could want for more?

Currently Reading

2004-10-10T20:28:32Z

I've been reading mostly fiction recently. I'd kinda stopped reading anything non-technical for far too long; there was always some new techie book to read, but there always will be... I used to read masses of fiction, mostly SF and fantasy stuff; but recently I'd only read fiction when on holiday, and then only as a backup to the techie stuff I had with me... That changed, on holiday, when I read Dan Brown's The Da Vinci Code. I loved it, bought the other three that he'd done and read those back to back. Angels and Demons was, IMHO, almost as good, Digital Fortress sucked (in the way that books that are about things that you have an understanding of, but that don't do the subject justice suck), Deception Point was OK but by that point I was starting to see the formula too much and the books were less interesting. I enjoyed Close to the Machine: Technophilia and Its Discontents by Ellen Ullman, though the "I'm getting old, I'm losing it, I don't even load all of the MSDN disks anymore" thing hit close to home ;) I enjoyed Zodiac, by Neal Stephenson and To Ride Hell's Chasm by Janny Wurts, but then I've been a long time fan of hers. The latest book was The Magicians' Guild (Black Magician Trilogy) by Trudi Canavan, which was pretty good fantasy; good enough for me to buy the second in the series...

Stepping back to reality, if this techie world in which I live can be called reality, I'm reading Hacking the XBOX: An Introduction to Reverse Engineering, which is an interesting looking book about hardware hacking, full of hints and tips on soldering and stuff ;) I've got several hardcore C++ books on order and I'm currently trying to work out which books I should take on The Trip, but more on that later.

Mythical Man Month

2004-05-20T20:22:55Z

I finished re-reading the Mythical Man Month today. It was a good read, and, just as when I read it just after the 20th anniversary edition came out, still very relevant to software development today.

I found Brooks' enthusiasm and attitude to be quite compelling; it's so obvious that he loves his work. This quote from the epilogue sums it up and reminds me not to be so grouchy; "To only a fraction of the human race does God give the privilege of earning one's bread doing what one would have gladly pursued free, for passion. I am very thankful." Me too.

One thing that struck me is that, whilst in general I agree with the central theme of No Silver Bullet, I think perhaps there are serious gains that can be made in many corporate programming shops just by removing all the unnecessary bureaucratic obstacles that they seem to delight in putting in the way of their programmers... Perhaps the corporates do have a silver bullet out there...

More than 20 years on and still as much to say

2004-05-09T17:25:21Z

I'm currently re-reading The Mythical Man Month 20th Aniversay Edition and I've just finished watching Apocolypse Now Redux. Both seem to have as much to say now about their respective subjects as they did in the day. Brooks' treatment of the "Joys" and "Woes" of The Craft as poignant as Coppola's tale of the lies and horror of war.

Currently reading

2003-07-22T20:24:40Z

Agile Software Development - Principles, Patterns, and Practices by Robert C. Martin

This book is physically heavier than most of the books I've been reading lately but I'm still carrying it to work even though I only get around 5 mins reading done on the tube during the journey. It's a beautiful book; the typeface and illustractions are stunning, the paper feels rich, the cover is cool and colourful.

The content is pretty good too. Robert Martin knows his stuff and this book seems to pull all of the things that I'm currently interested in together in one place. I'm only just getting into the book, but so far it's great and I'm impressed. I'll report back once I get to the end.

Waltzing with Bears

2003-07-07T06:27:57Z

Finally finished reading Waltzing with Bears: Managing Risk on Software Projects and it was well worth the read.

Waltzing with Bears is a book about managing project risk. It's a slim volume, but packed with useful information. As usual, DeMarco and Lister present the topic in an approachable and readable way. The text is full of anecdotes that flesh out the theories with practical examples.

In a nut shell; most project managers on software projects fail to adequately manage risk. In fact, most fail to even acknowledge risk exists. DeMarco and Lister suggest a more grown up method of managing projects; where risks are identified and tracked and generally allowed for in schedule and budget calculations. They provide a downloadable risk tool to help you run simulations of your project and produce risk diagrams showing how all your risks combine.

Lots of good stuff, but as I approached the end of the book I found myself thinking "It's all true, but it'll never work". They admit that in many corporate cultures it's not easy to publicly manage risk; especially if you're the only project manager doing it. Being anything but "Can Do" can be harmful for your career prospects in some places.

The final section offers some help though. It suggests that project sponsors need to specify the value of the project; and more specifically, the value of parts of the project. This value should be specified in terms that are comparable to the terms required in your scheduling and budgeting forecasts. If they want to say "But we just need it!", then you should be able to say "It'll take a long time and cost a lot of money!". If they want higher resolution then they have to specify the value to the same resolution... Doing so allows you to prioritise feature requests and bring the risk models to the table...

For project managers in large organisations I think it's likely that this book needs to have been read and accepted by people way above you before it's safe to openly practice what it preaches. However, if you read it at least you can start to push the message upwards.

Test-Driven Development (By Example) - Kent Beck

2003-05-27T06:57:11Z

Kent Beck demonstrates the testing side of XP by separating it out into its own simple methodology. Test-Driven Development is exactly what it says it is. The entire design and development effort is driven by the tests that you write and you're encouraged to write those tests first...

Parts one and two contain worked examples of Test-Driven Development. Some will probably say that they're too simple, but I've found that even the most complex domain usually ends up as relatively simple code if you develop in this way. It needs to be simple to be testable. One of the major advantages of writing tests as you write code is that you are forced to actually use your code very early on. If the interface is complex and hard to test then it's probably also hard to use; as you simplify to make the code easier to test you improve the design of the API itself.

As someone who develops in this style, albeit with far less tests than I should have and often by taking far larger steps than shown in the book, what I found interesting was the way the design converged on something that was 'right' as the code and understanding grew; this is definitely true to reality. I think, perhaps, that some people may find the coding steps shown in the book too small and this may put them off. It may have been worth trying to take far larger steps initially and then cut them down to size as the difficulty in taking larger steps became apparant. Although Kent makes it clear that you can take larger steps and write more code as your confidence grows I find that what usually works best is just to 'walk faster' in smaller steps.

My one complaint with this book would be that unless you've programmed in this style before it's probably not obvious exactly how fast it's possible to go whilst taking such small coding steps. Although there are some graphs of time taken between test runs and number of lines changed per refactoring these possibly don't help make it clear that the worked example is probably less than a day's work in all. I think it would have been handy to have some timestamps at each change to show just how quickly things can happen.

Part 3 of the book contains details of some patterns that are useful in Test-Driven Development. Things like Mock Object and Self Shunt are great tools when testing and once you've discovered the power of being able to plug in mock up versions of your code's dependencies to enable testing your designs will change forever. The result being loosely coupled objects that are easy to test and, as a nice side effect, easy to use and reuse.

A useful and interesting book. If you come away from it feeling that it can't possibly work, give it a try anyway, you'll be surprised.

Slack - Tom DeMarco

2003-05-23T06:12:55Z

I've always been a fan of DeMarco's work. I tend to nod my head and agree as I read, and wish that all the software development managers that I work with would read his books. Slack is an excellent analysis of the problems plaguing large corporations' software development efforts.

The book starts off looking at how and when knowledge work gets done; the myth of the fungible resource, and how the drive for efficiency in many corporations leads to everyone being so busy that they have no time for anything but the task at hand. Busyness replaces business. DeMarco addresses the costs of this pressure, the effects of 'aggressive schedules' and many of the other every day issues that paralyse these 'efficient' organizations.

The book then looks at change management and DeMarco comes to the conclusion that it's impossible for change to occur in organizations unless there is some slack. Highly 'efficient' organizations are less flexible and thus less effective than organizations where the people involved actually have a little time to think about what it is they do and how they could make it better.

We end up with a lightning tour of risk management; "Can Do" attitudes are directly at odds with proper risk management as to manage your risk you need first to accept that it exists. Can Do cultures don't allow this doubt to be aired publicly and so can't even begin to manage risk.

The book grabbed me, was easy to read and was one of those books where I felt the author was just confirming all the things I already knew to be true. The section on risk management is very approachable and gives some excellent ammunition for the next meeting that you have where people refuse to accept the existence of project risks.

Highly recommended.