Frans Bouma's blog

LLBLGen Pro v5.6 has been released!

Mon, 23 Sep 2019 07:01:47 GMT

(All LLBLGen Pro related blog posts are nowadays posted on the LLBLGen Pro blog, so I'll post here just a link to the actual post on the LLBLGen Pro blog.)
We released LLBLGen Pro v5.6! See this post for the details regarding what's new in this release!

LLBLGen Pro v5.5 feature highlight: query batching

Wed, 05 Dec 2018 12:20:00 GMT

(This is a stub, which redirects to the article on the LLBLGen Pro blog. I've stopped blogging LLBLGen Pro full articles here and instead do that on the new LLBLGen Pro blog. I post a small post here to notify readers the post is over there :) )

Insert/Update query batching is a new feature of LLBLGen Pro v5.5. This article will give insight in how this new feature can help you with high latency insert/update work.

To read the full article, please read the full article on the LLBLGen Pro blog.

LLBLGen Pro v5.5 feature highlight: Window functions

Wed, 05 Dec 2018 12:19:00 GMT

In LLBLGen Pro v5.5 we've introduced a new feature called Window functions support. What exactly are 'window functions' and why are they so important? To answer these questions let's first look at a good description of what a window function is.

To read further, please read the article over on the official LLBLGen Pro blog.

LLBLGen Pro v5.5 has been released!

Tue, 06 Nov 2018 10:00:00 GMT

After a two week beta period, we’ve released LLBLGen Pro v5.5! To see what’s new, please visit the New features page on our website or go to the beta announcement blog post which lists all the small changes in one place.

In the coming weeks, I’ll publish some articles regarding some of the new features to better highlight them and how they can help you in your projects.

Stay tuned!

LLBLGen Pro v5.5 beta has been released!

Mon, 22 Oct 2018 10:45:00 GMT

Last Friday we’ve released LLBLGen Pro v5.5 beta! We expect to release the RTM in a week or two. Below is the list of what’s new in this release. I’ll describe some of the new features in more detail in articles in the coming weeks. The main focus was on our own ORM framework, the LLBLGen Pro Runtime Framework, this time around.

What’s new in the LLBLGen Pro Runtime Framework?

New functionality / changes

Insert / update query batching (Adapter). The runtime now supports batching of insert and update queries into single DbCommand objects which great helps with performance, especially when using a database over a connection with high latency, like a cloud database. Available for Adapter, on the databases SQL Server, PostgreSQL and Oracle. More information
Window specification support (OVER() clause)). The LLBLGen Pro Runtime Framework now comes with support for the SQL OVER() clause, which is used with window functions. OVER() allows aggregates and specific window functions to define a window in the resultset. This support is two fold: existing aggregate functions can be extended with an OVER() clause in the SQL query and new window functions like RANK() and FIRST_VALUE() are supported and can be specified directly in an LLBLGen Pro query together with an OVER() clause. More information.
Union / Union All support in Linq and QuerySpec. The LLBLGen Pro Runtime Framework finally gets support for a feature that has been requested for many times: support for UNION and UNION ALL in queries. For Linq, there's now full support for .Union() and .Concat() (the Linq equivalent for UNION ALL). For QuerySpec, the support comes in the form of two new methods: Union() and UnionAll(). More information: Linq, QuerySpec
Oracle is now supported on .NET Standard. Oracle has finally released a .NET Standard 2.0 compliant ADO.NET provider, and you can use the LLBLGen Pro Runtime Framework now with Oracle on .NET Standard supporting platforms like .NET Core 2.x. More information
Function mappings of functions with no arguments for Linq. Mapping a function which has no arguments, e.g. GetDate() is now supported for Linq (QuerySpec already did support this). This allows you to map a .NET method which doesn't accept any database originating elements to a database function. More information
Query / Optimizer hints for Linq / QuerySpec. It's now possible to specify a query hint or optimizer hint to help a database optimizer with a choice to make when optimizing a given query. Available for SQL Server, Oracle and MySQL. More information Linq, QuerySpec.

Small changes

IEntityCollectionCore.Clear(bool) has been added, where if the boolean argument is set to true, the entities in the collection are added to the RemovalEntitiesTracker collection of the IEntityCollectionCore instance (if present).
An EntityView(2) no longer issues a Reset with the ListChanged event when a field of an entity in the view changes however it has no effect on the order of the entities in the view. It only issues a Reset with the ListChanged event if the order changed due to the change of the entity field.
The BooleanNumeric type converter now supports float and double values as well for boolean. This means that a conversion from boolean will result in 1.0 and a conversion to boolean will be using Convert.ToBoolean(value).
An implicit conversion between Enum and string is now supported, besides the already supported implicit conversion between Enum and a numeric value (int/byte etc.). This means you can map an enum typed field onto a string based database field without a type converter. The conversion will use the ToString() method on the Enum instance to produce the string value when persisting an entity to the database.
Implicit numeric conversions between fields of type byte, sbyte, short, int, long, float, double, decimal are now supported without a type converter.
Introduced object name creation caching to avoid string concatenations which are repeated a lot of times per query. This both saves memory and some query creation performance.
ISortClause now has two new command chaining methods: SetCaseSensitiveCollation(bool) and SetEmitAliasForExpressionAggregateField(bool) which set resp. the CaseSensitiveCollation and EmitAliasForExpressionAggregateField flags and return the ISortClause instance again for command chaining.
ISortClause and ISortExpression now have an extension method ThenBy() which accepts an ISortClause and which is appended to the object it's called on. The ThenBy() method is in the QuerySpec namespace.
The Oracle DQEs for ODP.NET and MS Oracle now have a static flag, DynamicQueryEngine.AlwaysUseRowNumBasedPaging, which is set to false by default. If true, and compatibility is set to 12c, a paging query will use the classic rownum approach instead of the FETCH NEXT / OFFSET approach. The latter can be up to 40 times slower than the rownum approach (see: the JOOQ blog) so the classic rownum approach might be more efficient in your application than the new approach.
The Firebird DQE now has function mappings for DateTime properties Year, Month, Day, Hour, Minute, Second, MilliSecond, DayOfYear, DayOfWeek and the DateTime methods AddYears, AddMonths, AddDays, AddHours, AddMinutes, AddSeconds, AddMilliSeconds.
When a retrieval query is forced to perform client side paging, distinct filtering or limiting, it will now append resp. "Requires client side paging", "Requires client side distinct filtering" or "Requires client side limiting" when the ORMQueryExecution tracer is enabled and the level is set to 'verbose' (4).

What’s new in the Designer?

Major changes / fixes / new features

Major performance improvements across the board, especially for larger models (250+ entities).
Support for implicit numeric conversions. The designer can now deal with implicit numeric conversions between model fields and mapped target fields, without the need for a type converter. This feature is supported on frameworks which allow implicit numeric conversions without a type converter: Entity Framework Core 2.1+ and LLBLGen Pro Runtime Framework v5.5+
Support for Entity Framework Core 2.1 has been added.

Minor changes / fixes / new features

The designer now supports implicit conversions between Enum and String. It already supported implicit conversions between Enum and a numeric value (int, short etc.). The implicit conversions are supported on target frameworks which support these implicit conversions as well. There's no value length check validation.
The TDL statement <[If HasTypeConverterDefined]> now returns true as well if the field mapping's property RequiresImplicitNumericConversionAtRuntime returns true, to make sure TDL can deal with this new feature without a new statement.
The TDL statement <[TypeConverterFullName]> will now also return the type converter name for implicit numeric conversions if no type converter is specified but the field mapping's property RequiresImplicitNumericConversionAtRuntime returns true. It will do so for C# and VB.NET.
PostgreSQL: The provider type for timestamptz fields is now TimestampTz instead of TimestampTZ. This change makes generated code require an Npgsql version of 3.2 or higher. This is done as TimestampTZ is marked deprecated and causing warnings during compilation. There's no change in functionality.
The designer now uses v18.1.6 of the DevExpress controls, which give a nice performance increase across the board.
The Visual Studio integration now hides the toolwindows if there's no longer a project loaded.
Several additional system type converters were added: ByteArrayStringConverter, DateTimeOffsetBinaryConverter, DateTimeOffsetByteArrayConverter, DateTimeOffsetStringConverter, DateTimeBinaryConverter, DateTimeInt64Converter, DateTimeStringConverter, StringByteArrayConverter, TimeSpanInt64Converter and TimeSpanStringConverter.

As always, v5.5 is free for people with an active subscription.

LLBLGen Pro v5.4 has been released!

Tue, 01 May 2018 11:35:38 GMT

Today we’ve released LLBLGen Pro v5.4! This release comes packed with a lot of small enhancements and some bigger ones. See below for the full list of changes. This update is a free update for all users with an active subscription.

Designer

Major changes / fixes / new features

PostgreSQL 10.x is now a supported database.
Support for PostgreSQL 10 identity fields has been added: in the PostgreSQL driver, designer and DDL SQL templates.
Addition of the CliSyncWithDatabase commandline tool, which is the re-incarnation of the CliRefresher for v5.x: it allows you to refresh catalogs (sync with a database) on the command line. Like with the CliGenerator, it requires a valid license to operate.
The code generator now caches regular expressions it's using and is optimized overall. It now offers faster template interpretation/execution (2x-3x faster code generation as in previous versions)

Minor changes / fixes / new features

The setting Code Generation -> Clean up vs net projects has been removed, as the setting is required to be set to true always and there's no advantage to ever have this setting set to false.
When a perform rule in a code generation task resolves to false (so the task is skipped), the remark in the log is no longer a warning, but has been changed into a normal remark.
SQL Server connection strings no longer contain 'packagesize' directives as the default value used isn't in sync with some defaults in the various .NET versions. This change means it now falls back on the default the .NET framework used is using.
When adding an entity to a derived model by right-clicking the entity and there's just one derived model in the project, it's automatically preselected in the selector dialog.
DTO/DocDB class models: the vs project files for .NET full no longer refer to a non-existing 'Properties' folder.
DTO class models: the projection func creation method now has a user code region so you can add additional projection fields to the projection.
TemplateBindingsViewer: When the templatebindings file is chosen in the combobox, the focus is automatically moved to the list of file bindings so scrolling with the mousewheel doesn't accidentally scroll the combobox but automatically scrolls the list of bindings.
It's now possible to generate DDL SQL scripts for a selection of catalogs instead of all catalogs in a database: select one or more catalogs in the project explorer or catalog explorer and select the DDL SQL script command you want to run from the context menu: the catalogs selected are now used for generating the scripts, the non-selected catalogs are not taken into account.
Several elements in the project explorer now offer edit actions for themselves, their mappings and their settings. For instance right-clicking a Field allows you to immediately go to that field's mappings in the entity mappings tab or to its settings on the Code Generation Info tab of the entity's editor. This provides an easy way to navigate from elements in the project explorer to their editors and detailed information.
Adding an entity with a value-typed field to a derived model as a new derived element now correctly unrolls the valuetyped field as a nested derived element.
The search/replace dialog for the text-editor is now embedded in the tab, like VS and Rider do: a panel with the search controls is opening when the user presses Ctrl-F and is closed when the user presses ESC or clicks the close button.
When tabbing away from the TemplateBindingsViewer or the PresetViewer and back, the control which had the focus is remembered, so when coming back to the still open templatebindings viewer tab or preset viewer tab, the focus is set on the control that had focus when you tabbed away.
If a project file contains multiple mappings for a single element to the same type of target for the same database (e.g. an entity is mapped onto two tables), the designer will report an error and list each mapping as suggestion to be removed.
Copying a TypedList with relationships into the same project now no longer creates dummy m:n relationships which could make the project unloadable

LLBLGen Pro Runtime Framework

Generated code is now much smaller, especially for Adapter. This leads to faster compilation times and faster code generation times (on top of the faster code generator in v5.4). A lot of code has been moved to the base classes EntityBase and EntityBase2. These changes have no negative memory impact nor performance impact, in fact memory consumption in prefetch paths is slightly better now.
The PlainSQL API now has a feature to perform implicit conversions of types, meaning a value of type T1 in the resultset can be implicitly converted to type T2 in the projection class, if an implicit conversion is defined for T1 and T2 (e.g. short to int)
Poco Typed List fetch methods in Linq and QuerySpec now have an optional query argument to be used as the root query, allowing you to specify a richer query than the generated query for the typed list.
PostgreSQL 10.x is now a supported database, which includes support for the new Identity fields in PostgreSQL 10.x
Empty methods for Field-mapped-onto-related-field event handlers are now avoided and code is less verbose
UnitOfWork and UnitOfWork2 now have a property IsEmpty which returns true if no work is scheduled in the unit of work instance. This property is also implemented in the interface IUnitOfWorkCore.
PostgreSql: the query generated for directly deleting entities using a join with another table now no longer uses non-ansi joins but uses a proper ansi-join using correlated subquery.
When fetching prefetch paths with a paging query, the runtime now automatically switches over to parametrized IN clauses for the prefetch path sub paths, no matter the value for the ParameterizedPrefetchPathThreshold. This avoids that all rows in the target of a sub path node are fetched and filtered on the client. This is now also done if a limit is specified for the root node and no paging is performed, the runtime in that situation now also picks the parameterized route.
The PostgreSQL DQE now uses the RETURNING clause to return sequenced fields in an insert query instead of a separate scalar query which used a separate roundtrip; each insert query is always one roundtrip, even if there are sequenced fields involved.
SelfServicing: DeleteMulti/UpdateMulti and async variants are now virtual.
Using FetchProjection<T>(retrievalQuery) or FetchQuery<T>(plainSqlString) where T is a valuetype like Int32 or a system type like string, the query would fail in previous versions but will now succeed with returning a list of values (the values of the first column in the resultset), all typed T.

Enjoy!

Let’s add a photo mode to Wolfenstein II: The New Colossus (PC)

Fri, 16 Feb 2018 13:43:04 GMT

Introduction
Getting started, the tools we need
What information do we need?
Setting things up
Memory allocation in windows processes
Finding the FoV
Finding the camera
Hide HUD using the g_showhud variable
Game pause / Timestop
What else is there to find?
Creating the camera tools
Conclusion

Introduction

I have a weird hobby: I like to take screenshots in games. Some call this activity game photography, and in a sense this is a bit like photography: composition, light, the things you have to deal with when taking photos in real life, are concepts you work with here as well. For photography you need a camera, and games in general present you with one, but there’s a problem: you aren’t going to be able to move and handle the camera however you please: it’s likely tied to the main character, there’s all kinds of info on the screen (Head Up Display or ‘HUD’) and the lens characteristics used are likely very wide (meaning: high Field of View (FoV) and low focal length).

To solve this, we need to alter the game so we can move and change the game’s camera however we want, but how to do that? It’s not as if games in general offer options for that. In this article I hope to explain a bit how to create a DLL we’re going to inject into the game’s process to take over the camera and control other game aspects for the purpose of taking screenshots. For my screenshotting hobby I needed ways to control the cameras in the games I played and so I started another hobby: creating camera tools so one can take proper screenshots in games without limits. I do that using my own system, which you can find at the link, which is open source software (C++ with a little bit of x64 assembler).

Sometimes a game offers a photo-mode which gives you control of the camera at times and lets you take screenshots within a given bounding box in the game. These are great if there’s nothing available, but in general they’re pretty limiting and the game we’re going to work with in this article doesn’t offer one so in cases where the photo-mode is pretty limited or non-existing, we have to add one ourselves.

This all works only on PC, not consoles. On consoles, you have to work with the photo-modes the developer offers you. In case you’re wondering whether it’s legal: strictly speaking reverse engineering is allowed (Compaq vs IBM). Additionally we’re not changing game code on disk, but strictly in-memory at runtime. We’re not adding cheats at all, even though some of the tools we’re going to use are designed for these. But it’s a grey area, and to avoid getting banned online, I have the strict policy not to create camera tools for online only games, but only for off-line single-player games.

The game we’re going to work with is the demo version of Wolfenstein 2: The New Colossus on PC. You can download it on Steam here. As the demo version is free, you can try everything discusses below yourself. The resulting tools also work on the full game, btw, so if you own the full game, you can use them there as well.

Disclaimer: I’m not a 3D engine programmer, I develop an entity modeling and object-relational mapper system for .NET as my job, but in a past long forgotten I’ve spent over a decade in the Dutch demoscene (mostly on Amiga 500) and have written a system called ‘DemoGL’ back in the early 2000’s (a system to create OpenGL based demos with, before the shaders took over) so I’m familiar with assembler, C/C++ and graphics. If you’re not familiar with these, that’s fine, you’ll still likely pick up a few things here and there .

Getting started, the tools we need

To get started we need some tools. After all, the game’s executable is 56MB and how are we going to find the code to tinker with so we can take over the camera? I only use free / open source tools and I’ll list them below. I know there are commercial tools available like IDA Pro, but as these tools cost a lot of money, we’re going to stick with free tools for now, as they provide the functionality we need.

Notepad++: We have to take notes and scribble things down, so let’s do that in the notepad tool which is way better than the one in windows
Visual Studio Community 2017. On the page click the link on the left. When installing, be sure to select the C++ related features and the Windows SDK and check this thread to get MASM enabled in your project, as we’re going to dabble with x64 assembler as well.
Cheat Engine. This tool is going to be our starting point in finding what we need to alter. It’s a tool which has a lot of features but is pretty arcane so you have to use it a bit to get used to its quirks. Be sure to exclude the crap tools it installs during the installation.
X64Dbg. This is an advanced debugger, assembler/disassembler and reverse engineering tool and I use this for the features that I need which I find cumbersome in Cheat engine or which aren’t available in Cheat engine.
Windows calculator in ‘programmer mode’.

These are the tools we need to get all information we need so we can then implement them in a C++ project to complete our camera tools. All tools / game are run as Administrator to avoid Windows blocking what we want to do (in general it won’t, but just in case).

What information do we need?

For a good photo-mode you need the following:

a way to move the camera freely, preferably with a controller besides mouse / keyboard
a way to rotate (pitch/yaw/roll) the camera freely
a way to change the lens length (resulting in a change to the Field of View or ‘FoV’) so we can zoom in/out
a way to pause the game so we can take our time to setup the shot
a way to hide any 2D HUD crap in the frame like ammo icons, mission directive icons and text

Those are the absolute basics. Additionally, things which are nice to have:

a way to scale the resolution so we can take a shot at a higher resolution than we’re playing the game at
a way to enable ‘depth of field’ to add bokeh blur
a way to change time of day, weather, colors

These ‘nice to have’ things are often complicated to find but if we’re lucky we might stumble upon them in our quest for the camera. If not, that’s all right, as long as we have the basics, we can take pleasing shots. For most games which use Direct3D or OpenGL we can use an effect library called Reshade which allows you to enable effect shaders like Depth of field or color adjustments. The game we’re working with uses ‘Vulkan’, and not Direct3D so we can’t use Reshade here.

So how do we find these basic elements? To save us a lot of time, we need to make sure we understand what we’re dealing with so we can search dedicated for a given value or range of values. So let’s start with the basics: what is a game camera?

The camera

As we’re working with a 3D game, we have a 3D world in which we can move around, and so does the camera. This area is called world space. Everything in this world has coordinates related to the world’s origin or (0, 0, 0), and so does our camera: it’s somewhere in this world at a given coordinate (x, y, z) (no worries, we’re not going to do complex math here… for now ). A game camera is essentially a point-camera: it’s a camera with infinite sharpness and has no size. It’s used to project the elements it sees in world space to a space called camera space. Camera space is effectively the area which is seen by the camera so it starts at the camera’s location (the infinitely small point in 3D world space) and it ends at the horizon:

(Picture from: https://www.ntu.edu.sg/home/ehchua/programming/opengl/CG_BasicsTheory.html. An excellent page if you want to know more about 3D rendering, it explains everything in simple to understand parts.)

At zNear is our screen we’re looking at. You’re at EYE, looking at the screen. The end of what you can see (often the ‘horizon’ in 3D world space) is at zFar. The cropped pyramid formed by this is called the ‘view frustum’. Everything in that frustum is shown on the screen.

How that works, that’s not really important. What is important though is that a camera in world space comes with a direction too. So it has a coordinate in world space and also a direction it looks at. In a 3D engine the camera coordinates combined with the direction it looks at and how it’s orientated (what’s ‘up’, what’s ‘right’) are bundled in what’s called a projection matrix. This matrix consists of 4 rows of 4 values (so a 4x4 matrix) and when you multiply it with a coordinate in world space the result is the coordinate in camera space. This neat feature is one of the many things your GPU is doing while you’re killing bad guys in the game.

Because of this, 3D engines calculate this projection matrix and use that for rendering the 2D image we see on screen and use it in all aspects of the rendering pipeline. To calculate this projection matrix, it thus has to know where the camera is in world space, and has to know how the camera is oriented (the direction it is looking and what is up/right). We’ll be overwriting this data with our own version of it so the game will use our camera as if it’s its own. This way we can control the camera’s location, direction and orientation without ruining anything inside the game.

You can store the look direction and orientation information in several forms: as a quaternion (a vector with a rotation), a 3x3 matrix (with the vectors lookat, up and right), angles (rotation around x, y and z) and I’m sure there are some other forms I’ve not yet seen. Most games use the quaternion variant as it is in practice the most reliable. Some engines use a 3x3 matrix (the engine we’re working with does too, so we’re going to have to alter a 3x3 matrix) and e.g. Unreal Engine 3 used packed Euler angles for rotation. In practice it doesn’t really matter: we’re going to use our own camera and convert it to what the engine uses.

In the picture above you see a distance between zNear and zFar. If we move zFar towards zNear, the angle FoVy changes: it gets bigger, or better: by changing FoVy, we change the distance between zNear and zFar. This results in things getting zoomed in/out. You can accomplish this in several ways: by moving the camera location towards zNear or by changing the angle FoVy. Most 3D engines opt for the second form and we’re going to exploit that information in our quest.

Our game uses a single value for FoV, it’s changeable in the menu. This isn’t the complete story: there are two angles, not one: one for x and one for y. However as you can calculate both from the single value given (combined with the aspect ratio) it’s easier to simply offer the user a single value to change instead of two and mess up the aspect ratio, as they’re all related to each other.

Game pause

Besides the camera information, we also need to pause the game whenever we want and manipulate the camera at that point. Every game has a way to pause itself (except the Dark/demon souls games and their lookalikes), often by pressing ESC or the menu button on the controller, or by simply alt-tabbing. In the early days when games were single-threaded simpletons, there was just one flag which made things stop. Nowadays with multi-threaded job-oriented 3D engines, a single flag isn’t likely going to cut it: various subsystems interact with each-other and stopping one needs to stop other systems too.

So even though we pause the game by pressing ESC and bringing up the menu, this is a multi-step, multi-value process. This sounds complicated but it can be straight forward: all we need is one value which stops at least one subsystem and we can start from there. For performance 3D engines in general use simple byte values (e.g. 1 is paused, 0 is not paused) as writing them is an atomic operation and this doesn’t require any locks. So if we find one of them, we can check what code reads / writes these bytes and see what else it checks.

This isn’t always successful. Some engines use a totally different way to pause the game, e.g. a float value which is the ‘speed’, a complicated function which returns 1 or 0 based on a packed bit field, pausing the job system by removing its pointer from a list, there are endless possibilities. For now we’re going to go for the easy way out: a byte.

Hide HUD and other on-screen cruft

To hide the elements of the HUD we have to find what code renders them so we can simply place a ‘ret’ statement at the start of it, or if the engine is using one, setting a byte to some value to indicate we don’t want any HUD elements. For the game we’re working with, it’s fairly easy: it supports a dev console! To enable it, go into steam, right-click the ‘Wolfenstein II: The New Colossus Demo’ entry and select ‘Properties’. Then open ‘Set launch options’ and specify:

+devMode_enable 1 +com_skipbootsequence 1

After this, when you start the game you skip the intro movies and you get a dev console in the game by pressing the ~ key (the key above the TAB key). The dev console allows you to type in commands which are executed immediately by the game engine. When you type:

g_showhud 0

followed by Enter, the HUD elements will all be hidden, and if you type ‘g_showhud 1’ (without the quotes) you’ll get them back. You can bind them to keys using the ‘bind’ command, but we’re going to use this little statement to find a 1 or 0 value in memory and set it ourselves so the user doesn’t need to open a console to hide HUD elements.

To recap…

We need to find: (float is 32bit single precision floating point value, double is 64bit double precision floating point value)

The camera coordinates (X, Y, Z), which are floats or doubles, but usually floats
The camera rotation/orientation, which is either a quaternion, a matrix or a set of angles. All of these are stored as floats most of the time.
The value which represents the FoV, which is usually a float
A way to pause the game, we’re going for a byte which is set to either 1 or 0
An integer value which represents the variable ‘g_showhud’ in memory, likely a byte.

We can combine this with some other information: most games are written in C++ with parts written in C. This means that they’re likely using classes or structs to store the camera data and the like together in memory. This is important because when you find one element like the FoV or the camera coordinates, the other elements we need are often close by as they’re in the same class/struct. We can use this to our advantage when we start looking for things. So let’s get started!

Setting things up

First we have to set things up so we can find what we’re looking for. This begins with starting the game. This particular game begins with a lengthy series of heart-breaking cutscenes which are unskippable. You have to sit through them till you end up as the main character in the wheelchair and a gun in your hand, preferably after the scene where you kill the first enemies.

As we have to switch between game and tools a lot, it’s easier to run the game in windowed mode with a low resolution, e.g. 1440x900 or smaller, the smaller the better (especially if you have just one monitor). Once you’ve arrived at this spot where you are controlling the player moving around and there are no enemies around, we can start our journey.

At this point we open notepad++ so we can add notes and we start Cheat engine. When you run Cheat engine, you have to attach it to the game’s executable by clicking the flashing computer icon on the toolbar, select the ‘Wolfenstein II The New Colossusx64vk’ process and click ‘Open’.

Before we proceed we first have to setup some hotkeys. In Cheat engine go to ‘Edit –> Settings’ and then ‘Hotkeys’. In the list of actions, click ‘Next Scan-Increased value’ and specify a hotkey, I use the key Ctrl-]. Do the same thing for ‘Next Scan-Decreased value’ and give it a hotkey, I use Ctrl-[ but really those hotkeys are up to you. It’s best not to use function keys as games often use these function keys too and you don’t want to activate ‘quick load’ when you actually wanted to continue a scan .

We’re now setup to find our first value. But first a note about memory allocation and code.

Memory allocation in windows processes

We’re about to search for our first value, however finding that value is just half of it. What we’re really after is the code which accesses that value. The main problem we’ll run into is that in modern OS-es, memory allocated by the process is placed at random locations (called ‘addresses’) in the 64bit address space. This means that if you find e.g. the FoV float value at address 0x7FF770620000, the next time you’ll run the game the value is likely on another address and you have to repeat the whole process again. However if you know which statement accesses it, you can set a breakpoint there and obtain the location again. This is also usable for our camera tools later on where we have to intercept code to obtain these addresses at runtime.

So once we find a value, it’s crucial we find the code accessing and writing to the address the value is at, so we can intercept it at that location at runtime at any time. We’ll see that Cheat engine helps us with that.

Finding the FoV

To find the camera, there are several ways to do that. Often one will use the ‘stair walking’ method, which exploits the knowledge that in 3D world space the ‘up’ axis is usually positive (usually Y or Z). So you can walk up a flight of stairs in-game, search for increased values, then walk down, search for values which are now decreased, and repeat that process till you have narrowed it down to a short list of values. You then block writes to these values, walk up / down the stairs again and if you blocked the right value, the screen will stutter (as Cheat engine will write the old value again and again). You can then narrow it down to the one value you’re looking for.

I find that method rather tedious and it’s not always needed: often a game stores the FoV close to the coordinates / rotation data and finding the FoV is way easier than walking up/down a flight of stairs as you first have to find one in the 3D world. In this particular game the protagonist is in a wheelchair (at least in the demo, which we’re using) so walking up/down some stairs is out of the question anyway.

Looking at the options in-game, under ‘Video –> Advanced settings’ we spot a ‘Field of View’ setting. We can also set this in the dev console by using the command “g_fov value” (without the quotes). To learn what the value is, go to the in-game options ‘Video –> Advanced Settings’ and see what ‘Field of View’ is set to, it’s likely either 90 or 72 or something like that. We can scan for that value using Cheat engine. Alt-tab to Cheat engine and go over to the right top corner where our scan controls are. We’re going to scan for a floating point value, namely the value we just learned from the Field of View setting. As Value Type, select ‘Float’, as Scan Type select ‘Exact value’ and specify the value to search for. My Field of View setting was set to 72 so I type 72. On the right you see some other options like rounding. I most of the time use Rounded (default) for floating point searches, but for this case either one is fine. Now click ‘First scan’. Cheat engine will scan the process’ memory for all values of 72 (or whatever your value was). It finds thousands of values. Oh boy, this looks like it is going to take while!

Well… not really. See, within this set of found values, we can scan again for a value. We’re now going to change the value FoV was set to in-game to a different value and see which ones have changed! It’s easier to do this in the dev console, so bring it up by pressing the ~ key and type “g_fov 65” (without the quotes) and press Enter. You’ll immediately see the game’s camera zoom in a little due to the smaller Field of View angle. Now we go back to Cheat engine by alt-tabbing to it. Change the value you have typed in before (I had 72 there) into 65 as that’s the new value the variable should have. Then click Next scan. Next scan will scan within the values we’ve found.

Lovely, the list of thousands of values has been reduced to just a couple. One of them is in green. Green values are values in writable memory pages within the executable’s image in memory. This might sound complicated but in general these are static variables and are allocated in memory within the executable in the process’ memory (the executable comes with blocks of memory which contain constants and the like and also static variables). The advantage of these values is that they’re always on the same spot: they’re part of the executable so every time we load the executable, they’re on the same location. While this might sound interesting, green values for cameras in general mean: they’re used as reference / fallback values (and likely here the value of the menu), but almost never for live camera values. So we’ll ignore the green one for now and select the other addresses with the value 65 and click the tiny red arrow in the lower right corner of the grid. This will add the two values to the lower part of the UI. It looks like this:

The addresses you’ve found are different from the ones in the screenshot above, which immediately illustrates why it’s important to find the code accessing these values. We now have two values and we can try to change them and see what that does. If I alter the first value by double-clicking the value in the value column, e.g. to 20, and then go back to the game, I see the camera has zoomed in a lot. The other value we’ve found has changed to the value 20 as well, so it’s a derivative from the first one. We clearly found the FoV!

Now we have to find out what code reads/writes to this value. Select the first value (the one which made the game zoom in when you altered the value) and press F5. This will bring up a popup stating you’re about to attach a debugger to the process. Click ‘Yes’ and Cheat engine will open a window which is empty. Alt-tab to the game and resume the game: you’ll now see one line and a counter that is increasing rapidly being added to the window that just opened. Now press the left-mouse button to bring up the gun (or left trigger on the controller, doesn’t matter) and then press the right-mouse button to aim. This will slightly zoom out the camera. You’ll see more lines are added to the window with counters that increased a bit and then stopped. Release the right-mousebutton. You can repeat this a bit, but nothing new is added. Alt-tab to cheat engine again and click ‘Stop’, but not Close(!), on the window.

We now know which statements read and write to that FoV value we just found. Select the first line in the window and click ‘Show disassembler’. This opens a new window of Cheat engine which is the memory viewer/disassembler. It’s an important window which will give us all kinds of information. Click a value in the bottom half (the hex view) and press Ctrl-9. It now shows the memory as floating point values. Go back to the main window and select our FoV value in the bottom grid. Press Ctrl-B. The memory viewer now shows the memory location of our FoV value. Let’s see what happens if we go into the game and change things. Alt-tab back to the game and resume. Bring up your gun and aim. In the memory viewer window you should see values light up in red (values which are changing/have changed). You clearly see the FoV value change and also a second one close by which gets the same value. This is the other value we have in the lower grid in the main Cheat engine window.

The top of the memory viewer window shows the executable in disassembled form. It’s a full debugger where you can set breakpoints. We’re going to set a breakpoint on the line which reads the FoV value. If you lost the line here, just go back to the window with the statements which accessed our FoV value and select the first one, then click ‘Show disassembler’. In memory viewer, press F5. This will make the line in the disassembler view green. This means a breakpoint is set on that line. Go back to the game and resume. Whoops, stuff seems to hang! Luckily it’s just the breakpoint we just set: the game’s executable has now been stopped at the statement we’ve placed the breakpoint on.

Alt-tab back to Cheat engine’s memory viewer. You’ll notice there’s no mouse pointer. This is because the game grabbed the mouse using a low-level windows function and still has the mouse in its possession, but as it’s stopped, it can’t free it back to windows and Cheat engine isn’t grabbing it either. Luckily things can be controlled by the keyboard as well so we can proceed. The element I’d like to show is the register view in the disassembler. The register RAX and RCX are light-blue, meaning they’re involved in the current statement. If you now press F8 you’ll step one statement at a time. You’ll see things are slowly changing based on the statements executed. F7 will step into a called function and F9 will continue execution.

Press F9. Cheat engine will continue execution but will immediately break on the same breakpoint. Do this a couple of times and you see the value for the register RCX changes every time. Press F5 to get rid of the breakpoint and press F9 to continue the game’s execution. As the address where the FoV value was read from is in RCX, this means that the game uses this code for multiple values. We can’t reliably use this to find our FoV back. We do notice however that the function reads the value from the address [RCX+0x18]:

NewColossus_x64vk.exe+9551E6 - 8B 41 18 - mov eax,[rcx+18]

Numeric values are in hex, so the address the FoV value is on, is the value in the register RCX + the offset 0x18. Go over to Notepad++ and store the above information as: “FoV offset is 0x18, address in RCX".

Back to the unreliable spot in the code to find back the FoV value. This is an important part: it’s essential you use code which is only used for the value you’re working on. If the code is targeting other values, forget it, the code isn’t usable. We’re now faced with a problem though: the only code which was called a lot which works on our value isn’t reliable, the other values are only called when the player aims. Go to the pop up window with the statements which access the value (with the title “The following opcodes … “) and click ‘Close’. We don’t need these anymore.

We have one more ace up our sleeves: the other FoV value. It’s close by to our real FoV value, finding that one means we found the other one too, it’s 0x10 addresses lower. So we’ll repeat the process with finding which statements write to the particular value: go to the main window in Cheat engine, select the second value and press F5 (or right-click it and select ‘Find out what accesses this address’). This gives us a window which already contains a line of code and a quickly increasing counter.

When we alt-tab to the game and resume, we see other lines being added. When we aim our gun in-game, it will add even more lines. Alt-tab back to Cheat engine and in the window with the statements, click ‘Stop’. Select the first line and click ‘Show disassembler’. This is clearly a different piece of code, perhaps this is more reliable. Repeat the process of setting a breakpoint at this line by using F5 in memory viewer and see if the value in the RCX register changes. (Spoiler: it does).

So now what? Well, the RCX register is changing a lot in the function we’re in based on other values. We could check whether the right one is passed in when the function is called. To do that, we first break again on the line which accesses our FoV value, and now we’ll step over the statement by pressing F8 and also over the ‘ret’ statement (which means ‘return’). When we do that we end up at another piece of code. Scroll up the code a bit and the line above the line you’re at contains a call statement. This is the call to the function which accesses our FoV value. As the FoV value is accessed with a statement using the RCX register (and an offset, namely 0x28, it’s the derivative value from the one at 0x18, remember?), we should check whether the RCX register is set to the right value prior to that call. If that’s the case, we can intercept the address of the FoV value right there.

A few lines above the call statement you’ll see a ‘lea’ statement which sets the RCX register to a new value:

   1: NewColossus_x64vk.exe+FE8C62 - mov rax,[rsi]   2: NewColossus_x64vk.exe+FE8C65 - lea rcx,[rbx+00004FC0]         ; reads FoV address in RCX   3: NewColossus_x64vk.exe+FE8C6C - lea rdx,[rbp+67]   4: NewColossus_x64vk.exe+FE8C70 - mov [rbp+67],rax   5: NewColossus_x64vk.exe+FE8C74 - call NewColossus_x64vk.exe+9551B0  ; calls FoV function

Click on the line below that line and press F5 to set a breakpoint on that line. It will immediately break. Pressing F9 a couple of times shows no change in RCX, it’s always the same. Checking whether it is the value we’re looking for, shows it’s the right value (the FoV value is read from offset 0x18 in the FoV accessing function). Copy the lines above the call in the disassembler view. Cheat engine will popup a dialog what to copy. Simply click ‘Copy’ and go to Notepad++. There you paste the lines of code, adding ‘FoV’ and other info like that RCX at the call statement contains the FoV address and if you hadn’t store the info the FoV is at RCX+0x18, you should do that now too.

Pfew, that was a lot! Let’s hope finding the camera is simpler!

Finding the camera

Now, not to make you feel depressed, but finding these values can take a long time and is often making you feel like you’re looking for a nail in a haystack. This includes finding the wrong values even though they look perfectly fine. To give you an example of that, I’m going to show you what that looks like. This particular game is a first-person shooter, so the camera is at the same location (roughly) as the player character: moving the player means moving the camera, there’s no way in-game to change the coordinates of the camera without moving the player. This gives of course the problem that when we find some coordinates, are these the coordinates of the player or the camera? This matters because you really don’t want to move the player, you want to move the camera, the player should stay put.

We’ve found the FoV and as I said the camera in general is close by, so we should check whether that’s the case here too. The easiest way to do that is by simply looking at the memory while we move the camera. Select our FoV value in the main window’s bottom grid in Cheat engine and press Ctrl-B. Make sure the display type is ‘Float’ by selecting a value and pressing Ctrl-9. Now go back to the game and resume and move the mouse around and move the player around a bit using the controller or WASD keys. We’ve learned that a camera consists of at least 3 values for its coordinates (X, Y and Z) and a couple of values which make up the rotation/direction of the camera. We don’t see that much values change here, so the camera might be further up in memory (which is down in the memory viewer ).

Looking at the piece of code I copied above, the line with ‘here’ on it contains an offset, and a rather big one. ‘lea’ means that the address in the register RBX + the offset specified is loaded into the register RCX. This means that the value for the FoV is at offset 0x4FC0 in the struct / object which starts at the address in RBX. In theory this could be a big massive object with a lot of data which we have to look at to see whether there are coordinates / matrices and the like. That’s quite a lot of ground to cover. But if things are easy, everyone would be doing it, right?

Let’s start at the address in RBX at that location. To get that address, we place a breakpoint at the lea rcx, [rbx+0004FC0] statement. The game breaks immediately. Now click on the value besides ‘RBX’ in the Registers view and copy the value from the dialog that pops up and close the dialog. Now right-click a value in the bottom half of the memory viewer and select ‘Goto address’ and paste the copied value into the dialog and click OK. The memory viewer should now move to the address you’ve pasted into the dialog. It sometimes switches back to another address, so keep an eye on that.

Now press F5 to remove the breakpoint and F9 to continue executing. The memory doesn’t really reveal much, lots of zeros, not much interesting there. Scroll down a few pages and you should see a couple of values grouped together, in the –1.0-1.0 range and values like in the range 6.0-15.0, depending on where you are in the game of course. Scroll these into view and go back to the game.

Moving the camera around should make the area in memory look like this:

A lot of values close together change. Moving the player around also changes the values which are bigger than 1.0. Rotation matrix values are always between –1.0 and +1.0, coordinates of course can have any value. This might be the camera data we need although it looks suspicious: the camera coordinates are part of a full 4x4 matrix (as they’re at the bottom of the matrix), and as camera coordinates are often used separately as well, the code reading these values would look rather awkward in C/C++. Nevertheless, it’s a set of values that changes whenever we move the camera, so it’s worth investigating.

You see at least two values which are larger than 1.0. Right-click the first one (in the screenshot above it’s the 6.63 value) and select from the context menu “Add this value to the list”. A dialog pops up, for description type in ‘X’ and press Enter. If you now look at the bottom grid in the Cheat engine main menu you’ll see that the address of the value is now added to the grid. We can now check what code access that value. Select the value we dubbed ‘X’ in the bottom grid of the Cheat engine and press F5 (or right-click and select ‘Find out what code accesses this address’. A window pops up but it’s empty.

Going back to the game and resuming gameplay, gives only one line, a write. This is clearly not the camera we’re looking for, there’s no code reading it, at least not this moment. However it must be a copy from somewhere and as the value’s changed when we change the camera in-game, the origin where the value comes from might be the right camera. Go to the window with the single line of code and click ‘Stop’ then select the line of code and click ‘Show disassembler’. This brings you to a piece of code which clearly copies values from one place to another. Often these are copy constructors of C++ classes, and when you see them, it’s a sign the value you’re looking at isn’t the right one, but the origin might.

Looking more closely at the code, we see that values from the location in RAX are copied to the location in RSI. So we need to look at the memory RAX points at. Press F5 to set a breakpoint at that location and go back into the game and resume. This will trigger the breakpoint, but there’s again no mouse pointer. Press F9 again to continue execution and we have to type the address manually from the register listing. Right-click in the bottom half with the memory values and select ‘Goto address’. In the dialog, carefully type the value that’s next to the RAX register in the Registers list at the top right corner.

If you type the address correctly, you’ll end up in a block of memory where you see three coordinates (likely X, Y and Z) and a matrix following it, likely a 3x3 matrix. This looks more promising. Right click the selected value (the first value at the top left corner) and select ‘Add this address to the list’ and give it the description ‘X2’. After you’ve removed the breakpoint from the disassembly listing in memory viewer, you can now verify whether this is data that changes when you’re moving about in game, you’ll see that the values light up red (so do a couple of values which look remarkably the same close by however).

Go to the Cheat engine main window, bottom grid and select X2. Then press F5 to see what code accesses this value. The window that pops up immediately gets two lines of code with increasing counters. Going back to the game and resuming gameplay adds a whole lot more, some with very fast increasing counters making the game stutter a bit. Alt-tab back to cheat engine and click ‘Stop’ on the window with the statements and counters. All statements are reads (in Intel assembly, statements have the general form of <opcode> <destination>, <source>) except one. That one is interesting, as we can now replace it with no-operation (nop) statements so the write is effectively not there anymore. Find the line in the list which looks like this:

7FF61050A04C - F3 0F11 AB 90FF0000  - movss [rbx+0000FF90],xmm5

Right-click it and select ‘Replace with code that does nothing (NOP)’. Now go back to the X2 value in the main cheat engine window’s bottom grid and double click the value to change it. If it’s e.g. 6.2, change it to 16.2. Go back to the game and you’ll see the camera has moved to another area. This looks promising indeed! However does the game think the player is there too? Sadly, it does. If you, in-game, press the left-mouse button a couple of seconds the gun will fire and you’ll fire bullets right there where the camera is. The player model isn’t there though, but the game clearly thinks the player is at that location. We actually found a player location but not the camera. In first person games this often happens, and all we can do is scratch this and move on.

A quick note on replacing code with code that does nothing: you have to be careful what you replace: sometimes data is overwritten by the memcpy function which is in the mvcrt dll or other memory copy function. Replacing that statement will replace it for the complete game and you’re modifying the code which is used a lot. This in general leads to a quick crash of the game and you can start over. So in general check whether the statement is inside the exe or game dll (Cheat engine shows the module/file the address is in in the first column of the disassembly view) and if not, ignore it entirely as it’s not what you’re looking for anyway.

In the main Cheat engine window, on the bottom-left you’ll see ‘Advanced options’ (I did warn you the tool is a bit arcane ). Click it and a window pops up. You’ll see the line of code you just replaced with ‘NOP’ statements is there in red. Red means that line is replaced by something else. Right-click it and select ‘Restore with original code’. If you now go back to the game, the player object and its model in-game are re-united and everything is as it should.

No camera though. You can look for it further up in this memory block but I can save you the trouble: it’s not there. So where is it? What can we do now to find it? This is the point which makes creating these tools so much fun: you have to think out of the box, find tricks and schemes to find what you want, to make the game give you the data you’re looking for. Luckily for us, we still have some information left to explore: the two FoV values used for fovx and fovy, calculated from the one FoV value the user can tinker with.

To find the locations of values in cheat engine (which simply scans the process’ memory for the value specified) you either have to know the exact value or have a way to change the value in-game so you can search for changed values and narrow it down. We have an easy way to change the FoV in-game: aiming the gun. So let’s exploit that here.

We’ll start a new scan on the Cheat engine main window. If you still have the scan data from a previous scan, click ‘New scan’. If the button says ‘First scan’, just continue with setting up the scan: This time we don’t know the initial value (as the game might store this in radians or in degrees) so we’ll pick for Value Type ‘Float’ and for Scan Type ‘Unknown initial value’. Then click on First scan and cheat engine will scan the complete process’ memory for floats. It will happily reports it found over a billion different values.

Great.

Now we’ll go back into the game and we’re going to use the hotkeys we’ve defined earlier. Resume gameplay and bring up the gun using the left-mouse button or the controller’s right trigger. When we aim the gun (by using the right-mouse button or left trigger on the controller), we see that the game zooms out a bit. Zooming out means the FoV value increased. This might sound counter-intuitive (as zooming out with a lens means shortening the focal length), but think about it this way: zooming out means there’s more in the view frustum to see so the frustum is wider and higher, meaning the angles increased. Now hold the RMB to keep aiming (or the controller’s left trigger) and at the same time press the hotkey for Next scan-increased value. I have defined Ctrl-] for that, so I press and hold RMB and then press Ctrl-]. Cheat engine now performs a new scan within the results of the last scan and only keeps values which increased in value.

Now release the RMB / left trigger so the aiming action / zoom in-game is gone, now press the hotkey for Next scan-decreased value. I have defined Ctrl-[ for that so I now press Ctrl-[. You can repeat this process a couple of times and the list gets smaller quickly, however you’re still stuck with hundreds of thousands of values. To cull that list even more, we can alt-tab to cheat engine and perform a next scan for the scan type ‘Value between’. Fill in as values 0.1 and 100 and click Next scan.

This will cull the list to a couple of thousand values. Repeat our aim / not aim steps a couple of times and we’re left with a thousand or so values. If you look at the list of addresses found, it’s a long list of addresses close together, sometimes a new block starts but all in all they’re long lists of addresses which are very similar. For now we’ll skip those and scroll till we don’t see long lists of similar addresses which is all the way to the bottom. This gives you two addresses with the values in the range of 35-70 which are not green and a couple of green ones. We’ve learned to ignore the green ones for camera values and therefore select the two addresses right above the green ones which have values close to 40 and close to 65-70. Mine were 39.4 and 66.4, but it depends on the aspect ratio of the window, so yours are likely different, but nevertheless in that range. Once selected click the red arrow to add them to the list.

Select one of them and press Ctrl-B to view their place in memory and the values around them. Go back to the game and move the camera around. Red values change all over. We’ve found what looks like a camera: coordinates and a 3x3 rotation matrix. Right-click the first value of the changing values (likely in the range of 6-10 if you’re still at the start of the game) and select ‘Add this address to the list’ and give it a description ‘CamX’. Do this for the two values next to it as well and name these CamY and CamZ. Now go to the Cheat engine main window and select CamZ and press F5 to see what code accesses this value. Go back to the game and resume gameplay. Nine lines of code are added with fast increasing counters. Alt-tab back to cheat engine and click ‘Stop’ on the window with the statements and counters.

They’re all reads except one, the one which looks like this:

7FF610508ED9 - F3 0F11 87 F0000000  - movss [rdi+000000F0],xmm0

This statement writes the Z value. Right click that line and select ‘Replace with code that does nothing (NOP)’. Then go over to the Cheat engine main window, double-click the value for CamZ and change it to a slightly higher value (e.g. from 0.9 to 2.9). Now go back to the game and resume gameplay. Whoa, you’re at the ceiling, but the player character isn’t. Pressing the LMB till the gun starts shooting reveals the game thinks the player is still at the same location as it was before but the camera clearly isn’t. This might be the camera we were looking for, but before we can draw that conclusion we have to verify whether the code that writes isn’t used by other code, so it doesn’t write to other addresses as well.

Click the line above in the window with the statements (“The following opcodes… “) and click ‘Show disassembler’. You’ll see the list of ‘nop’ statements instead of the actual statement because we’ve replaced it with code that does nothing, remember? Right-click the top ‘nop’ in the disassembly view and select ‘Restore with original code’. Looking at the code we can see that the data is coming from a structure on the stack (RBP is the bottom of the stack). When you pass a structure by value in C/C++ the structure is copied onto the stack (if it’s not too big), so the calling code has calculated the new values and placed them on the stack so they could be stored at this location. Place a breakpoint at the statement we just restored by pressing F5. The game breaks immediately. Pressing F9 to continue executing breaks the game again, but RDI doesn’t change, the destination is always the same.

Bingo! We found our camera .

So the first thing to do is to copy the code we are at to Notepad++. Remove the breakpoint, press F9 to make the game continue executing and select all these lines from the disassembly and copy them over to Notepad++:

   1: ;Address                     - Opcode bytes          - Disassembler   2: NewColossus_x64vk.exe+FE8EC2 - F3 0F10 45 C7         - movss xmm0,[rbp-39]   3: NewColossus_x64vk.exe+FE8EC7 - F3 0F10 4D CB         - movss xmm1,[rbp-35]   4: NewColossus_x64vk.exe+FE8ECC - F3 0F11 87 E8000000   - movss [rdi+000000E8],xmm0  ; Write X   5: NewColossus_x64vk.exe+FE8ED4 - F3 0F10 45 CF         - movss xmm0,[rbp-31]   6: NewColossus_x64vk.exe+FE8ED9 - F3 0F11 87 F0000000   - movss [rdi+000000F0],xmm0  ; Write Z   7: NewColossus_x64vk.exe+FE8EE1 - 0F10 45 E7            - movups xmm0,[rbp-19]   8: NewColossus_x64vk.exe+FE8EE5 - F3 0F11 8F EC000000   - movss [rdi+000000EC],xmm1  ; Write Y   9: NewColossus_x64vk.exe+FE8EED - 0F11 87 F4000000      - movups [rdi+000000F4],xmm0 ; Write matrix value 0-3  10: NewColossus_x64vk.exe+FE8EF4 - 0F10 45 F7            - movups xmm0,[rbp-09]  11: NewColossus_x64vk.exe+FE8EF8 - 0F11 87 04010000      - movups [rdi+00000104],xmm0 ; Write matrix value 4-7  12: NewColossus_x64vk.exe+FE8EFF - F3 0F10 45 07         - movss xmm0,[rbp+07]  13: NewColossus_x64vk.exe+FE8F04 - F3 0F11 87 14010000   - movss [rdi+00000114],xmm0  ; Write matrix value 8  14: NewColossus_x64vk.exe+FE8F0C - 48 8B CF              - mov rcx,rdi

As there’s just one write (that we know of at this moment), we can now build a basic camera and intercept it. But we’re not done yet, we still have to hide the HUD and find the game pause. So before we’re going to develop camera tools, we continue with our research.

Hide HUD using the g_showhud variable

The next piece of information we need to find is to disable the HUD elements and other 2D stuff that’s rendered on top of the 3D scene. We have some help in this case, as there’s a handy cvar we can set on the dev console: g_showhud. If the cvar has the value 1, HUD elements are shown. If it’s 0, everything (including crosshairs) are hidden. We could leave it at that and force the user to make sure they have the dev console enabled and bind some hotkey to this cvar but that’s a bit of a hassle. On top of that, in general devs don’t really enable these engine cvars in release builds, so it might very well be that the publisher (Bethesda) will update the game and remove all access to these cvars, making the camera tools useless.

Finding this variable should not be that hard: What we have to do is setting the variable in the dev console a couple of times and use Cheat engine to search for a byte value being either 1 or 0. To find this value, start a new scan in the Cheat engine main window: for Value Type select ‘Byte’, for Scan Type select ‘Exact value’. To prevent you screaming at the screen for 10 minutes because Cheat Engine seems to hang: never and I mean never start a new scan for the value ‘0’. Cheat engine searches the whole x64 memory space available to the process and most addresses have the value 0. Searching for 0 will therefore lead to a lot of data and Cheat Engine has to write this in a temp file on disk. Even with an SSD it will take a lot of time and it’s unnecessary: start your scan for the value 1 instead of 0.

To make sure we scan for the right situation, alt-tab to the game and press the ~ key to bring up the in-game console. Type “g_showhud” (without the quotes) and press Enter. It should say “1”. If it says “0”, type “g_showhud 1”. We now know the HUD is enabled and the variable we’re looking for has the value 1. Alt-tab to Cheat engine and specify for value the value 1 and click ‘First scan’. It finds billions of values. Alt-tab to the game and in the dev console type “g_showhud 0” (without the quotes) and press Enter. Alt-tab back to Cheat engine and fill in for value the value 0 and click ‘Next scan’. Click it a couple of times, you should have a lot less values now, likely 50K or less.

Alt-tab back to the game, type “g_showhud 1” (without the quotes) and press Enter. Go back to Cheat engine, fill in for the value the value 1 and click ‘Next scan’ to scan within the found values. You likely have a thousand or less values. Click ‘Next scan’ a couple of times to further limit the list (after all the value didn’t change so we can weed out false positives which changed while we’re using Cheat engine). Repeat this process again: set g_showhud to 0, go back to Cheat engine, specify the value 0 and click ‘Next scan’, then set g_showhud to 1, search for 1 using Next scan, do this a couple of times till you have a small list of only 50 or less addresses found.

After a couple of further scans, you’ll end up with just one address. It’s a green address (NewColossus_x64vk.exe+2F960E0), which means it’s a static variable and this time we’re going to use it. Values like this are often defined statically (which means it’s a global variable) and it’s not unreasonable to see this variable being defined as global here too. The beauty of a static variable is also that its location is always the same. We can’t really depend on that however as we’ll learn later because we want our tools to survive a game update if there is one (which might change addresses where things are located, including static variables).

So to be on the safe side, we still need to see the code which reads this address so we can use that to find the variable’s address reliably in our camera tools. Select the address we found and click the red arrow to add it to the bottom grid of the main Cheat engine window, select it and press F5 to make Cheat engine show what code accesses this value.

When going back into the game and resuming gameplay you’ll see 4 lines of code show up in the opened window with quickly increasing counters. Alt-tab back to cheat engine and click ‘Stop’ on the window as we no longer need to track access to the value. The 4 lines are the same: cmp dword ptr [7FF6BAF160E0],00. This statement compares the value on the given address with 0. But… it checks a 4 byte value, we searched for a single byte. As the other 3 bytes are 0 it doesn’t matter, but if we had searched for 4 bytes instead of 1 we might have found this value quicker. It doesn’t really matter which line we’re going to copy to our notes, so pick the first one, click ‘Show disassembler’ and copy a couple of lines around it (10 or so) and copy them over to the notes in Notepad++, and specify after the cmp statement that it reads g_showhud there.

Game pause / Timestop

We now have almost all basic ingredients for a good photo mode: FoV, camera control and a way to hide the HUD. There’s one element left: to pause the game whenever we want. This is a tough nut to crack however. As I explained earlier: game engines have a lot of different options to implement a ‘pause’ in the engine, and as several subsystems work together in parallel, it’s often a multi-step process. The first thing to try is to see whether we can find a byte variable which is 1 when the game is paused through e.g. the menu or by opening the journal and 0 when we’re in the game playing.

It’s key we have a shortcut setup for ‘New scan, exact value’. The reason being that we want to start the scan when the game window is active, even if we’re in the menu: some engines shut down almost all internal processes when you alt-tab away and therefore searching for a byte flag as we’re trying to do won’t work. Go into Cheat engine main menu –> Edit –> Settings and under ‘Hotkeys’ define a key combination for New scan – Exact value, this is different from Next scan-exact value, as New scan starts a new scan. I use Alt-=, but you can pick anything as long as you avoid function keys.

In the search area on the main window in Cheat engine, click ‘New scan’ if you just have scanned for a value. Specify for Value Type ‘Byte’ and for Scan Type Exact Value. For value specify the value 1 and alt-tab to the game and you’re likely in the menu. Now press your hotkey for New scan-exact value. It will find millions of values. Alt-tab back to Cheat engine and specify for value the value 0, then alt-tab back to the game and resume playing. Now press the hotkey for Next scan-exact value. I have the hotkey Ctrl-= defined for that. It will limit the list a lot. Look around with the camera, move the player a bit, and press Ctrl-= again a couple of times, the list will shorten.

Alt-tab to Cheat engine, now specify as value 1, go back to the game, and while the menu is still up, press your hotkey for Next scan, exact value. These steps should be repeated a couple of times, be sure you scan for 1 when the menu is up and for 0 when the menu is down and you’re in the game. You can undo one scan in Cheat engine, so be careful or you have to start over (it happens to everyone ).

After a while you are greeted with a list that’s a couple of hundred values and it won’t get much smaller. So we have to investigate what these values do, what they’re used for. If we think a bit about what we’re searching for, a variable that’s 1 if the game should be paused, and 0 if the game should be running normally, code reading / writing the variable should compare it to either 0 or 1. If there are just statements writing a 1 or a 0 to it, or it’s never read when the menu is up or the game is playing normally, the game won’t react to a value change when we ourselves set it so if that’s the case, we can skip them.

This was a bit of a nasty exercise, I’m sorry, as the results won’t result in what we’re looking for: this particular build of idTech 6 doesn’t have a byte flag for game pause. I’ve tried everything to find it, using the menu, using the journal, scanning for byte, for a 4-byte dword, but none would lead to a variable we can set which makes the game pause. So we are back to square one.

Reverse engineering and what it is all about

This is a good moment to go into what we’re doing here: we’re reverse engineering a 3D game engine with little to no info and we try to understand what various pieces of code do by investigating and researching things at runtime or by reading the code without running it. The more information we have, the more success we’ll have with finding what we’re looking for and understanding what we’re dealing with. As we’ve reached a bit of a dead end with our quest to find the game pause, we need more information to perhaps look for a different element, for a different piece of data, or perhaps we can just find the right piece of code.

The game we’re working with is built with ‘idTech 6’, the same engine used in the game ‘Doom (2016)’. What if we try to grab some information from that game and try to use it on the game we’re working with? I happen to have made camera tools for Doom, and found two timestops there, as well as some photo mode related data. Additionally I found a list of ‘cvars’ online for idtech 6 which contained variables like g_freezeTime and g_stopTime. Trying these out in the game’s console leads to nothing because they’re not supported in this build from the console. But perhaps the code for these variables is still in the game’s engine.

Doom to the rescue

Using Doom, I use the byte scan technique we’ve used a couple of times to find the ‘Photo mode enabled’ setting. This leads to a single variable which is used in the following code:

   1: ; Address          - Opcode bytes      - Disassembly   2: DOOMx64.exe+79A57B - 48 8B 10          - mov rdx,[rax]   3: DOOMx64.exe+79A57E - 48 8B C8          - mov rcx,rax   4: DOOMx64.exe+79A581 - FF 92 B8000000    - call qword ptr [rdx+000000B8]   5: DOOMx64.exe+79A587 - 48 8B 48 20       - mov rcx,[rax+20]   6: DOOMx64.exe+79A58B - 83 79 30 00       - cmp dword ptr [rcx+30],00         ; photomode read   7: DOOMx64.exe+79A58F - 0F84 59010000     - je DOOMx64.exe+79A6EE   8: DOOMx64.exe+79A595 - 83 BE 9CEA0000 0E - cmp dword ptr [rsi+0000EA9C],0E   9: DOOMx64.exe+79A59C - 0F84 4C010000     - je DOOMx64.exe+79A6EE  10: DOOMx64.exe+79A5A2 - 48 8B 0D 27DE5403 - mov rcx,[DOOMx64.exe+3CE83D0]  11: DOOMx64.exe+79A5A9 - 48 8B 01          - mov rax,[rcx]  12: DOOMx64.exe+79A5AC - FF 90 80070000    - call qword ptr [rax+00000780]  13: DOOMx64.exe+79A5B2 - 84 C0             - test al,al

This code can be very useful: we can search for the Opcode bytes which are the bytes the processor reads and which form the actual statements and perhaps the same statements are used somewhere in our game so we can start from there. To do this, I use a different tool, X64Dbg. Cheat engine can search for bytes but it often doesn’t find something while it’s definitely there, and X64Dbg is just never fails and is faster at this as well. The downside of X64Dbg is one you’ll soon run into: it will break on exceptions (errors) thrown by the .exe you’re debugging. This is annoying as a lot of executables throw a lot of errors at times especially when you alt-tab back into them. Debugging in Cheat engine is therefore easier because of that.

Start X64Dbg and in the File menu click ‘Attach’. A dialog pops up with all the processes available: click the NewColossus_x64vk.exe process and click ‘Attach’. As we’re attaching a debugger to a running process, Windows will notify the process of this event and an exception will be caught. To proceed, select Debug –> Run (Swallow exception). This should make the debugger continue and you can continue with the game. We now have to locate the actual exe in memory. To do that go to the Memory map tab (6th from the left). Scroll almost all the way down till you see in the ‘Info’ column our executable: newcolossus_x64vk.exe. Now right-click the first row which says “.text” in the info tab (see the screenshot below) and select ‘Follow in Dump’. Focus will change to the Dump 1 tab in the lower left corner on the CPU tab. This is the memory at that location in hex. It’s the code in byte form.

We’re interested in the following statements:

   1: DOOMx64.exe+79A587 - 48 8B 48 20   - mov rcx,[rax+20]   2: DOOMx64.exe+79A58B - 83 79 30 00   - cmp dword ptr [rcx+30],00      ; photomode read   3: DOOMx64.exe+79A58F - 0F84 59010000 - je DOOMx64.exe+79A6EE

To find these statements in our game’s executable in memory, we’re going to copy the bytes and replace any offset with ??. This gives us the byte string: 48 8B 48 ?? 83 79 ?? ?? 0F 84. We can skip the offset after the bottom conditional jump statement. Now click on the first byte in the ‘hex’ area in the Dump 1 tab and press Ctrl-B. A dialog pops up to allow us to search for a pattern. Copy the string of bytes above (48 8B 48 ?? 83 79 ?? ?? 0F 84) and paste them in the bottom area below ‘hex’ and click OK. X64dbg will now scan for this code and will show all places it finds it in a tab under ‘References’. It finds 1 reference!

Double-clicking it brings us to the CPU tab, in the disassembler overview of the code we just found. X64Dbg is quite a lot more dense than cheat engine and it overwhelms you with information which might be a bit too much for some, but stay with me, we’re almost there . Press F2 to set a breakpoint on that line. Now alt-tab back to the game and resume. It will hit the breakpoint in X64dbg. Alt-tab back to X64Dbg, move the mouse to make the debugger grab it back from the game and now step over the statement with F8. We’re now on the cmp dword ptr [rcx+30], 00 statement. RCX now contains the start of the structure our variable is in. On the right top corner you see the list of registers, click the value next to RCX and press Ctrl-C. We now have to continue the game’s execution. Click the line with the breakpoint (the red address) and press F2 to toggle it off. Now press F9 to continue execution.

We’re now going to Cheat engine as it offers something we need: binding a hotkey. In Cheat engine go to memory viewer and right-click in the bottom half and select ‘Goto address’. In the dialog that pops up press Ctrl-V and add ‘+30’ (without the quotes), e.g. you now should have: 00007FF7C8A196E0+30, as the value is at offset 0x30. You should view the memory as bytes, so right-click a value and select Display Type –> Byte (Hex). The variable is in the top left corner and is 0.

Right-click the value in the top left corner in the memory view and select ‘Add this address to the list’. For description we’ll give it ‘g_stopTime’ and click OK. Now go to the main window of Cheat engine and right-click the value we just added and select ‘Set/change hotkeys’. Click ‘Create hotkey’ in the dialog that pops up. Under ‘Type the keys you want to set the hotkey to’ type the ‘*’ key on the numeric keyboard. Click the ‘Toggle freeze’ value and select ‘Set value to’. Below it specify the value 1 and click Apply.

Click Create hotkey again and as hotkey specify Ctrl-*, again ‘Set value to’ and as value 0. When the two hotkeys are added, click ‘OK’. You can now control this variable with the hotkeys we defined. Alt-tab to the game and resume playing. Now press the hotkey numpad-*. The game should freeze but you can still move the camera. Neat, we found our timestop/game pause! Before we move on, we have to copy the code which references this variable so we can find it again, even if it’s a static variable. We still have the code open in X64Dbg so we can copy that, or we can select the variable in the bottom grid on the Cheat engine main window and press F5 to see what code accesses this variable. Add the copied lines to our notes in Notepad++ and specify that it’s the game pause variable.

If you do that, you’ll see a lot of compare with 0 statements, that’s always a good sign for a game pause variable. If you copy code (and thus use it later on) which refers to a register, we have to intercept the code there to read the register’s value to be able to write to it. If you copy code which contains the hard-coded variable’s address, it’s possible to grab that address directly from memory and we don’t have to intercept the statements, this is a little easier.

“Dude, this is just plain luck”

Of course! We barely know anything about this large 56MB big executable and we’re just prodding about to see what effect these changes have. If we see a change and it’s something interesting, why not use it? If nothing happens or the game crashes, we had bad luck and we have to start over. That we lucked out with the variable from Doom is great, it saves us a lot of time reverse engineering code which works on this variable. Sometimes you luck out a bit, sometimes you don’t.

What else is there to find?

Now that we have our game pause, and I already gave away it’s the g_stopTime cvar, can we find other variables as well? Large 3D engines often use some sort of scripting / command system for e.g. debugging or scripts for gameplay or to be used in the console. idTech 6 is no exception to this. These commands are stored as plain text strings in the executable and we can try to find code which references these strings to see where that leads us.

In X64Dbg, go to the CPU tab where we still have our code visible and right-click in the code area and select Search for –> Current module –> String references. X64Dbg will now search the complete exe for strings and will dig up all code that uses these strings. After a couple of seconds focus moves to the References tab where a subtab is added with the strings. On the bottom you see a ‘Search’ textbox. Type “g_stopTime” (without the quotes) and X64Dbg will limit the list to the entries which contain the string we typed. Double click the first “g_stopTime” in that list. X64Dbg now switches to the CPU tab where it shows the code using that string reference. Let’s look at this a bit closer

On the highlighted line, we see the address of our game pause variable. This piece of code apparently defines and initializes the command, as it refers to a help string as well and an error string. While the statement on the highlighted line is odd, it’s related to the value a couple of lines above in the ‘lea’ statement above the call, which is 0x30 smaller. That offset sounds familiar .

Using the help string we learn that if we set the variable to 2 everything stops, also the player movement. You can test this out right away if you want: go back to Cheat engine, select the variable in the main window of Cheat engine, right-click it and select ‘Set/Change hotkeys’. Add another hotkey, e.g. Shift-* which sets the variable to 2. Now go back to the game and resume and press Shift-*. This should be different from when you pressed *. If you now press * you can rotate the camera but not move. Ctrl-* brings everything back to normal. This also proves the variable we found is g_stopTime.

If we go back to X64Dbg and scroll up a bit, we see another cvar, g_runFrames, with almost the same code: the variable name, a small help string, an error string, and at the bottom a call and the same kind of statements as in the screenshot above. It’s now easy to grab the address for g_runFrames: double click the line: mov dword ptr ds:[7FF7C8A18340],esi, and the dialog that pops up contains the address, so you can easily select it and copy it. You should by now be able to navigate to this address in Cheat engine’s memory viewer (Right-click –> Goto address, simply ctrl-v as it’s the right address).

This variable, g_runFrames is great actually, as we can use it to skip a number of frames when the game’s paused. This would be a neat extra feature for the camera tools. Now go through the steps in cheat engine to add the address of the variable to the main screen grid of Cheat engine (right-click in memory viewer on the top-left value, select ‘Add this address to the list’) and try to determine which statements access this variable (Press F5 once you’ve selected the value in the grid on the main screen in Cheat engine, then go back into the game). Don’t forget to click ‘Stop’ when you found the addresses on the window as otherwise Cheat engine keeps a breakpoint open and these come in limited supply. There’s just one statement accessing g_runFrames but a hotkey bound to it, combined with pressing the earlier created hotkey numpad-* proves it still works: if we e.g. set the variable to 30, the game will skip 30 frames and then pause again. Select the line in the window and click ‘Show disassembler’. Then copy a few lines around it and paste it in our notes in Notepad++.

Scanning the list of strings some more we locate g_freezeTime. This is another game pause variable but it freezes everything. Optionally you can perform the steps obtaining its address (as it’s the same code structure as g_stopTime above), and code utilizing it. None of these variables is set by the menu or by opening the journal, so no wonder we missed them!

As we now have all the information we need, it’s time to build the actual tools. We could research some more to find the DoF settings, perhaps resolution scaling if it’s implemented (it’s in the menu, but tinkering with a value higher than 1.0 gives garbage on screen so some investigation is required) but those are pretty advanced topics and this article is already reaching a massive length so we’ll leave those for later.

Creating the camera tools

To create the camera tools, we’re going to use my Injectable Generic Camera System (IGCS), which is an open source library which contains all the features we need. To get started I usually copy an existing one and modify the copied source, instead of writing one from scratch and link to a .lib file. I opted for this setup because every camera requires a lot of customization and there’s always a feature or two which are supported in other cameras but not this one, or which are unique for this particular camera so linking to a .lib would be a bit of a burden, as it has to be kept backwards compatible with older cameras as well. In the future when things are settled down and not a lot of features are added anymore I’ll make it more generic and a universal lib.

For the camera tools we’re going to create here we can re-use the logic in the Doom camera tools I’ve made some time ago as it’s the same engine, e.g. for rotation order and what’s up: Z or Y and what’s right: X or Z or Y. However the Doom camera tools are pretty old: it doesn’t use Array Of Byte (AOB) scanning to find the code to intercept but instead uses hard-coded offsets. Therefore we’re going to re-use the camera tools for Hitman 2016 instead and modify that one to make it fit for Wolfenstein 2. The Hitman 2016 camera tools don’t use an overlay (as we can’t use one, as the game doesn’t use Direct3D so we can’t hook into Dxgi’s Present function) but still contain the more advanced calculations for addresses and the like so we use that as our base. As there are quite a lot of tiny changes and migrating the Hitman 2016 camera from vs2015 to vs2017 format, I’m not going to document them all here. What I will do is try to explain how IGCS works and how we use the information we gathered during our research in IGCS so the compiled dll is usable with our game and does what it must do.

TThe final camera sourcecode is available in the IGCS repository so you can download that and use the code with the article to get a better understanding what all parts of IGCS do. To compile the camera source in Visual Studio 2017, you have to set up Masm, the Microsoft Macro Assembler. Opening the camera tools .sln file in Visual Studio gives you a project node in Solution Explorer with the name ‘Wolfenstein2CameraTools’. Right click that node –> Build dependencies –> Build customizations and check the checkbox in front of ‘masm’ and click ‘OK’. Then in the tree, under ‘Game specific’ right-click ‘Interceptor.asm’ and select Properties. Under ‘General’ change Item type to ‘Microsoft Macro Assembler’ and click OK. The project should now compile.

Configuring IGCS to hook into the .exe

Camera tools built with IGCS are a dll, and are injected with an injector into the .exe at runtime. The injector is an open source tool (or you can use Extreme Injector if you want) and is part of the IGCS code base. It’s a simple command line tool but does the job. Once the camera tools dll is injected into the executable at runtime, Windows will call the DllMain function of the dll and will pass a value indicating what happened. This DllMain function is found in the Main/Main.cpp file. We’re only interested in Process Attach and Process Detach events. Process Attach is happening when the dll is injected and the process attaches to the dll: the dll becomes part of the address space (is mapped into the address space of the process) of the exe’s process and the dll can now reach everything the exe can too. The DllMain function is fairly simple: it creates a new thread which executes the function MainThread. This function first obtains information about the host process (the game’s exe), then starts up the IGCS system and then waits till the system ends. This way our code is kept alive as long as the game runs.

IGCS hooks into a couple of things: RawInput to obtain controller / keyboard / mouse input, the game exe itself to obtain addresses of where the camera is and other variables at runtime and if necessary the Dxgi Present function to show an overlay (this isn’t supported here, the game doesn’t use Direct3D). A hook is adding a detour at the start of an existing function in memory by changing its code: it first jumps to a function we provide (the detour function) and then if we want, we call the original function’s code, or we can simply return there, and the original function is never called. This hook functionality is provided by a clever library called MinHook.

(Raw)Input hooks

To be able to hook RawInput we need the window which for our process receives the raw input data. To find this window we simply need to specify the window title. In our game that’s “Wolfenstein II The New Colossusx64vk”. We specify that in the file Game Specific/GameConstants.h in the constant GAME_WINDOW_TITLE. This file contains other game specific constants as well and we’ll adjust them for Wolfenstein 2, like the version and game name. To be able to intercept XBox controller input, we need to hook the XInput dll’s functions. We do this in Hooking/InputHooker.cpp, where also the Windows message hooks are located. There are two versions of XInput, xinput1_3 and XInput9_1_0, which one to hook?

To find that, we can use Cheat engine. When Cheat engine is attached to a game’s process you can find all dlls the game’s exe references at runtime. Go to the memory viewer and click View->Enumerate dlls and symbols. This gives a tree view of all dlls and the functions/symbols these dlls offer. The game we’re working with, Wolfenstein2, however references both. This is fairly odd, as one is enough. Doom used 1.3, so for now we’ll hook xinput3_1. To do that, we have to specify the name of the library to hook on line 165 in InputHooker.cpp.

Game exe hooks

For hooking into code in the game exe itself I use a different method: I set the jmp statements myself and jump back to a specific location in the .exe. The main reason is that in almost all cases I need to intercept a specific piece of code in the middle of a function, or in a piece of code that’s linked with jmps and it’s unclear what the function is. Hooking these pieces of code with MinHook is difficult as there’s no function known to intercept, it can’t find a function to hook.

The hook is a simple jmp qword ptr [0] followed by the 8 bytes forming the address to jump to. As we’re in 64bit land here, we have to use this setup as all other assembler instructions for a jmp can’t reach potentially further than a 32bit range: it might very well our camera dll is more than 2GB in address space away from the exe. Using a jmp qword ptr [0] instruction solves that. It simply says to the CPU: jump to the address at the offset 0 from this instruction (meaning: right after it).

The problem is: the jmp instruction + the 8 bytes of the address to jump to is 14 bytes we need to overwrite. It might very well be the bytes we’re overwriting contain multiple statements (a CPU executes bytes, as the bytes form the code) and some jmp statement elsewhere jumps right in the middle of the block of statements we’re overwriting. If we write our address there (which can be anything so the bytes could form invalid commands or anything else) the jump will end up on those bytes likely causing problems, perhaps even crash the game.

So we can’t write our jmp instruction everywhere, we have to make sure no code jumps to that location. This is easy to see in X64Dbg. To avoid having to work with the live exe running, we will work with a dump of the exe instead. To create one, we have to have the game running and we start X64Dbg. In the menu select Plugins –> Scylla. Scylla is a nifty tool which can dump almost any process’ memory without being noticed. In the dialog select the game’s exe in the list of processes and simply click the Dump button. It will ask you where to write the dump .exe file, by default it picks the same folder as where the normal exe is located. After the dump has been created, we can load this dump exe in X64Dbg. One thing to keep in mind: the absolute addresses of the code in X64Dbg in the dump differ a bit from the ones in Cheat engine or when you attach the debugger directly to the exe. I haven’t been able to find a way to make these equal. It’s just an inconvenience as you have to recalculate the addresses if you need them, but as we’re working with relative stuff anyway for the most part, it’s not that big of a deal, just something to keep in mind.

The hooks IGCS sets in the game’s code at runtime all have the same structure: they jump to a small piece of assembler in the camera tools, all located in the file Game Specific/Interceptor.asm, where e.g. values from registers are written to variables of the camera tools (like an address in a register), and/or small pieces of code are skipped when the camera is enabled (writes to the camera matrix for instance), after which the assembler will jump back to the game exe to a pre-calculated offset. To find the exact spot where to write the jmp, we’ll use Array of Byte (AOB) scanning. AOB scanning is fairly simple: as an executable is simply a large block of bytes, as statements are strings of bytes, we can simply search for a unique string of bytes which is relative to the location of the code where we need to write the jmp.

In general it’s a good idea to use the bytes which make up the code we have to intercept/hook. We also have to make sure the strings of bytes are unique, so the string of bytes we’re looking for doesn’t match with more locations in the exe (IGCS’s AOB scanner can work with multiple locations, you can specify e.g. to use the 3rd one, but this is fragile). X64Dbg is better suited for this and the dump we just made of the process is good for finding these strings. So let’s see how this works with the code which writes the camera data. We can use that piece of code from the game exe to intercept the address where the camera data is located and at the same time block writes to that camera data when our own camera is enabled (so the data we’re going to write there isn’t overwritten by the game).

I always make notes like this:

   1: ;Camera:   2: ;Address                     - Opcode bytes          - Disassembly   3: NewColossus_x64vk.exe+FE8EC2 - F3 0F10 45 C7         - movss xmm0,[rbp-39]   4: NewColossus_x64vk.exe+FE8EC7 - F3 0F10 4D CB         - movss xmm1,[rbp-35]   5: NewColossus_x64vk.exe+FE8ECC - F3 0F11 87 E8000000   - movss [rdi+000000E8],xmm0    ; WRITE X. Intercept here   6: NewColossus_x64vk.exe+FE8ED4 - F3 0F10 45 CF         - movss xmm0,[rbp-31]          ;   7: NewColossus_x64vk.exe+FE8ED9 - F3 0F11 87 F0000000   - movss [rdi+000000F0],xmm0    ; WRITE Z   8: NewColossus_x64vk.exe+FE8EE1 - 0F10 45 E7            - movups xmm0,[rbp-19]         ;   9: NewColossus_x64vk.exe+FE8EE5 - F3 0F11 8F EC000000   - movss [rdi+000000EC],xmm1    ; WRITE Y  10: NewColossus_x64vk.exe+FE8EED - 0F11 87 F4000000      - movups [rdi+000000F4],xmm0   ; WRITE Matrix values 0-3  11: NewColossus_x64vk.exe+FE8EF4 - 0F10 45 F7            - movups xmm0,[rbp-09]         ;  12: NewColossus_x64vk.exe+FE8EF8 - 0F11 87 04010000      - movups [rdi+00000104],xmm0   ; WRITE Matrix values 3-7  13: NewColossus_x64vk.exe+FE8EFF - F3 0F10 45 07         - movss xmm0,[rbp+07]          ;  14: NewColossus_x64vk.exe+FE8F04 - F3 0F11 87 14010000   - movss [rdi+00000114],xmm0    ; WRITE Matrix value 8  15: NewColossus_x64vk.exe+FE8F0C - 48 8B CF              - mov rcx,rdi                  ; Continue here

This means: I want to intercept the code at the statement with the comment ‘Intercept here’ and I want to jump back to the game to the statement with the comment ‘Continue here’. I have to make sure there are at least 14 bytes between them (there are as you can see) and that there aren’t any jmps to code in the middle of it, as it’s detoured in our own version of this code in Interceptor.asm:

   1: movss [rdi+000000E8],xmm0 --->; replace with jmp qword ptr [0] to function in Interceptor.asm   2: movss xmm0,[rbp-31]          3: movss [rdi+000000F0],xmm0    4: movups xmm0,[rbp-19]         5: movss [rdi+000000EC],xmm1    6: movups [rdi+000000F4],xmm0   7: movups xmm0,[rbp-09]         8: movups [rdi+00000104],xmm0   9: movss xmm0,[rbp+07]         10: movss [rdi+00000114],xmm0   11: mov rcx,rdi              <;--- Interceptor.asm function jumps back to this statement

We now have to select a string of bytes from the set of opcode bytes starting with the point where we want to intercept the code. In general it’s best to avoid offsets in jmp statements and call statements. Luckily we have just mov statements in our codeblock. The offsets from the register RDI are the offsets in the struct the camera data is part of and this isn’t going to change much in patches. The other offsets, from the register RBP can change, as RBP is a stackpointer (the bottom of the stack). It’s best to replace these with ?? in the byte string.

To test our byte string, we use the same technique as we used to find the code we found in the Doom exe which led us to g_stopTime: follow the .text memory block in Dump in X64Dbg and using the pattern search using Ctrl-B. Using the byte string F3 0F 11 87 ?? ?? ?? ?? F3 0F 10 45 ?? F3 0F 11 87 ?? ?? ?? ?? 0F 10 45 leads to 1 location in the exe. Double clicking it brings us to the code in the CPU tab in X64Dbg and the code is indeed the right block. If you scroll it a bit down you see on the left of the code all kinds of arrows. Those are jump lines where code jumps to another place in memory to continue execution. No arrow ends in the block we’re intercepting, so we’re all good!

IGCS defines AOB strings as constants in Game Specific/InterceptorHelper.cpp, in the initializeAOBBlocks function. There it defines for every AOB string to find a string of bytes, binds it to a key (so we can display some text and refer to it), and then searches the string in memory. The address it locates the string on is stored in the AOBBlock instance. In the initializeAOBBlocks function, we’re specifying the string of bytes from the previous paragraph to the constant CAMERA_ADDRESS_INTERCEPT_KEY. The AOB scanning in IGCS has another trick: if you place a ‘|’ character in the string of bytes, the byte following that character is the address stored in the AOB block instance. This can be helpful if you need to specify some bytes in front of the bytes on the location you’re after to make the match unique.

The function setCameraStructInterceptorHook in Game Specific/InterceptorHelper.cpp is the one which sets the hook in the exe for the camera address intercept function, using the address found with the CAMERA_ADDRESS_INTERCEPT_KEY AOB block. It specifies a couple of parameters which are important, let’s look at them below:

   1: GameImageHooker::setHook(aobBlocks[CAMERA_ADDRESS_INTERCEPT_KEY], 0x40,               2:                   &_cameraStructInterceptionContinue, &cameraAddressInterceptor);

It passes the AOB block with the address to set the hook (the jmp statement), the offset from that address where to continue execution in the main exe. It then passes two addresses, namely the address of the variable to store the continue address in (as the value is used in the function in Interceptor.asm: it will jump to that address to continue), and the address of the function in Interceptor.asm which is the address the jmp statement written over the game code will jmp to. The offset specified here, the 0x40, is calculated from the offset values besides the .exe name in the listing from the notes: the value where to continue (0xFE8F0c) minus the value where to start intercepting (0xFE8ECC), resulting in 0x40. The Windows calculator in ‘programmer’ mode (in Hex view!) helps here.

So how does our function cameraAddressInterceptor in Interceptor.asm look like?

   1: cameraAddressInterceptor PROC   2:     mov [g_cameraStructAddress], rdi       ; store the camera struct in the variable   3:     cmp byte ptr [g_cameraEnabled], 1      ; if our camera is enabled, skip writes   4:     je exit   5: originalCode:   6:     movss dword ptr [rdi+E8h],xmm0         ; original code from the game, intercepted   7:     movss xmm0,dword ptr [rbp-31h]   8:     movss dword ptr [rdi+F0h],xmm0   9:     movups xmm0,xmmword ptr [rbp-19h]  10:     movss dword ptr [rdi+ECh],xmm1  11:     movups xmmword ptr [rdi+F4h],xmm0  12:     movups xmm0,xmmword ptr [rbp-9h]  13:     movups xmmword ptr [rdi+104h],xmm0  14:     movss xmm0,dword ptr [rbp+7h]  15:     movss dword ptr [rdi+114h],xmm0  16: exit:  17:     ; jmp back into the original game code, which is the location after the original statements above.  18:     jmp qword ptr [_cameraStructInterceptionContinue]  19: cameraAddressInterceptor ENDP

After the setHook function has run for the camera address intercept, it will write a jmp qword ptr [0] on the address the first movss statement is on which jumps to this function, cameraAddressInterceptor. The camera address struct in memory is in the register RDI. We store it in the variable g_cameraStructAddress. We do this every time because the location of cameras can change at any minute: your character dies and you have to reload, or you load a different level, or the game uses a different camera e.g. in a cutscene. This way our camera is always written to the right location. Not shown here but visible in the Interceptor.asm source is that I also always copy the notes I made in comments above the function so it’s easier to see what original code was used, where it was intercepted etc.

As the code we’re intercepting also performs the writes for the coordinates and rotation matrix, we have to disable those when our own camera is enabled, to avoid our own values being overwritten. We can do that by simply jumping over them and not executing them at all. A careful reader has noticed the statements in the ‘original code’ are slightly different: there are ‘h’ suffixes behind every numeric value and e.g. xmmword ptr keywords. These are necessary because the Masm assembler otherwise doesn’t work properly: all numeric constants are seen as decimal, not hexadecimal. 7 decimal is equal to 7 hexadecimal, but 12 hexadecimal is equal to 18 decimal, so things could lead to serious problems.

The other interception I’ll describe in more detail here is for the FoV variable address. The FoV read code was on a terrible location with a lot of conditional jumps. As we’ve seen earlier, we analyzed that the address in RCX was the one we needed and it was already set at the start of the function. So all we needed to do was looking at the code calling that code where it set the RCX register and intercept that. That’s done on the code below. This code was also mentioned earlier but is copied here for convenience.

   1: ;FoV Read   2: ;Address                     - Opcode bytes      - Disassembly   3: NewColossus_x64vk.exe+FE8C62 - 48 8B 06          - mov rax,[rsi]           ; Intercept here   4: NewColossus_x64vk.exe+FE8C65 - 48 8D 8B C04F0000 - lea rcx,[rbx+00004FC0]  ; Load FoV Address   5: NewColossus_x64vk.exe+FE8C6C - 48 8D 55 67       - lea rdx,[rbp+67]   6: NewColossus_x64vk.exe+FE8C70 - 48 89 45 67       - mov [rbp+67],rax       7: NewColossus_x64vk.exe+FE8C74 - E8 37C596FF       - call NewColossus_x64vk.exe+9551B0        ; call FoV function; Continue here

Avoiding any offsets, we can use the byte string 48 8B 06 48 8D 8B ?? ?? ?? ?? 48 8D 55 67 48 89 45, which leads us to the location of the code above. Checking the block in X64Dbg shows there’s no jump to a statement between intercept and continue, so we can use this block. It’s quite tiny but it’ll fit: remember we’re going to write 14 bytes of data. If there are less than 14 bytes between the intercept and the continue location we’re going to overwrite the statement we’ll jump back to so that won’t work. Just a quick reminder: I made all these mistakes and more and still do from time to time , no sweat if you don’t get it right the first time.

Our interception function looks like this (in Interceptor.asm). We have to copy this much code because we have to write 14 bytes.

   1: fovAddressInterceptor PROC   2: originalCode:   3:     mov rax,[rsi]   4:     lea rcx,[rbx+00004FC0h]          ; Read the FoV address in rcx   5:     lea rdx,[rbp+67h]   6:     mov [rbp+67h],rax   7: exit:   8:     mov [g_fovStructAddress], rcx    ; Store the address in the variable   9:     ; jmp back into the original game code, which is the location after the original statements above.  10:     jmp qword ptr [_fovAddressInterceptionContinue]  11: fovAddressInterceptor ENDP

The hook for this function is set in the function setPostCameraStructHooks in InterceptorHelper.cpp, which is called after the camera address has been found. In general you’ll need to intercept more code snippets than these, as in some engines a lot more code is writing to e.g. the camera addresses. However intercepting them isn’t hard, they work the same way as the ones described above.

For the variable g_runFrames, which we’ll be using in combination of the variable g_stopTime, we use the same technique as for the FoV: we intercept the original code which reads the variable’s address from memory (so the pointer to the variable) in a register and we store this value in a variable ourselves, then continue the code as if nothing happened. There’s a problem though:

if we simply intercept the code the actual line is on which reads the g_runFrames variable, we don’t have enough bytes to place our jmp qword ptr [0] statement on: it will overwrite the conditional jump below it or we need to take the call below it into account. This is difficult, as the call below it uses RIP relative addressing (using an offset relative to the address of the statement, see below for more details), and copying this code to our detour function in Interceptor.asm will require us to calculate this offset into a real address, adjust our own code, which is more work than necessary. We can however use more lines above the read statement, as long as we are sure we copy all lines which are involved in the small jumps in the code, see the small arrows on the left. If we start on cmp rax, r8 and continue on test edx, edx, we have all the space we need and can intercept the address without problems. This is why it’s important to check these jmp lines to see you’re not overwriting something with your hook which is actually a statement some other statement jumps to.

The full function to intercept g_runFrames then becomes (the addresses are corrected with local labels c1 and c2):

   1: runFramesAddressInterceptor PROC   2: originalCode:   3:     cmp rax,r8                               4:     jne c1   5:     lea rax,[rcx+000001F0h]   6:     jmp c2   7: c1:   8:     call rdx   9: c2:  10:     mov rcx,[rax+48h]  11:     mov edx,[rcx+30h]  12: exit:  13:     mov [g_runFramesStructAddress], rcx    ; Store the address in the variable  14:     ; jmp back into the original game code, which is the location after the original statements above.  15:     jmp qword ptr [_runFramesAddressInterceptionContinue]   16: runFramesAddressInterceptor ENDP

Setting up IGCS to write its camera data

We now have code in place which intercepts the exe on the right locations and which will give our own code the right addresses for FoV and camera data. Well…. almost. The addresses we intercepted are start addresses of structs/objects. The variables we’re after are located inside those structs, on a given offset. For the camera coords these are the offsets used with the RDI register in the intercepted code. For the FoV we have to look at our notes and see that the actual FoV variable is at offset 0x18. I always define constants for these, they’re in GameConstants.h, at the bottom and have names like CAMERA_COORDS_IN_CAMERA_STRUCT_OFFSET.

All camera / other data read/write code is in Game Specific / CameraManipulator.cpp. When the camera is enabled, it first backs up the current coordinate values, rotation matrix values and FoV value. This is done in the function cacheOriginalCameraValues. Its counterpart is called restoreOriginalCameraValues and is called when the camera is disabled. In theory not really necessary (disabling the camera enables the writes) but it’s good practice to clean up after you trashed the place.

IGCS’s main loop, which is located in the file System.cpp and is in the function mainLoop, calls every loop iteration a function which will update the current frame, which will end up in the function writeNewCameraValuesToCameraStructs which by default terminates if the camera is disabled. When the camera is enabled however, it will calculate a new rotation vector, a new movement direction vector based on the movement commands received via controller or keyboard and use those to calculate new camera coordinates. This information is then written to the camera location in memory in the function writeNewCameraValuesToGameData in the CameraManipulator.cpp file. Based on the game engine’s coordinate system and what rotation system it uses (e.g. matrix or quaternion) we have to perform different calculations, but in general the code is roughly the same.

Internally IGCS uses a quaternion system for the camera. This has the advantage that it’s always the same, no matter the engine. The quaternion is the direction in which the camera looks and also its orientation. This quaternion is built by creating three quaternions from the three axis and for each axis (X, Y or Z) the angle the current rotation rotates around that axis and multiplying them in the right order. The angles for the axis are the ones which are changed by the user using the mouse, keyboard or controller. My system isn’t the most optimal, in theory I could use the previous quaternion and update that one, however the approach I have is fairly common and works great so for simplicity’s sake I picked that one (also because I’m not a mathlete and quaternion math isn’t… simple).

The rotation / coordinate calculations are in the Camera class, namely in the calculateLookQuaternion function to calculate the quaternion to produce the new rotation matrix, and the calculateNewCoords function which calculates the new coordinates of the camera based on the current ones and the movement in x/y/z multiplied with the direction the camera is looking at (which happens to be the quaternion we calculated with the calculateLookQuaternion), so we move relative to where the camera looks and at different speeds. Rotation calculation is tricky, as rotating around x, then y and then z is different from rotation around z, then y then x for example.

The coordinate system of the engine used in the game is leading here: if for instance X is facing into the screen, the rotation around X is equal to ‘tilt’ and has to use the angle for tilt, plus it then has to be multiplied last (so it’s applied first). Note that the code uses DirectXMath.h for its calculations, which is the OSS 3D math lib for Direct3D, even though the game uses Vulkan and is clearly based on OpenGL. This works because the math 3D rendering is based on is universal, the different APIs just use different axis orientation and matrix layout (which are stored in memory the same way, btw).

So how does this all work together?

IGCS’ mainloop

After IGCS’ system has been started in the thread created in DllMain, it will initialize the hooks like RawImput, message interception and the hook to intercept the camera address. This is done in Initialize in System.cpp. The initialize method will then wait till the game calls the piece of code we’ve hooked to jump to our own cameraAddressInterceptor function in Interceptor.asm (see above). When that happens, e.g. you load a level in the game, the camera address variable will be set and the initialize function of IGCS will know the camera has been found and will proceed with setting the rest of the hooks like FoV, game pause and other hooks we might need.

After that’s all done, it simply runs an infinite loop in mainLoop in System.cpp. Every loop iteration it will first handle the user input (e.g. controller, mouse, keyboard input) and then write the new camera coordinates/rotation data to memory if the camera is enabled, otherwise it will skip that step. User input like controller stick movement or button presses will update rotation angles and direction movement data every frame. It also makes sure that the user can enable / disable the camera (through the keyboard) and can perform other actions like pause the game before the camera is enabled. It’s a simple system but very effective.

Now that we have our hooks setup for the camera and FoV we can build a dll and test it out.

Testing the camera tools

We compile the project in Visual Studio by pressing Ctrl-Shift-B. This will give us (if everything is OK) a dll. Unless you’re going to ship this dll, it’s best to compile a debug build (which can be selected in the Visual Studio toolbar). A debug build will write extra information to the console which can be very helpful if things don’t go as planned. To test the dll, we first run the game, and then run an injector to inject the dll. You can use the IGCSInjector if you want, or e.g. Extreme Injector if you can find the download online. I’ll use the IGCSInjector. I configure the IGCSInjector.ini file like the following:

   1: [InjectionData]   2: Process=NewColossus_x64vk.exe   3: Dll=Wolfenstein2CameraTools.dll

This requires that the IGCSInjector.exe, the IGCSInjector.ini and the Wolfenstein2CameraTools.dll are in the game’s root folder where the .exe is also located. Then first run the game normally and after it has opened the window, you run the IGCSInjector.exe. You’re greeted with a console window which looks like this:

Here it tells us that all AOB blocks were found successfully and all hooks were set properly. Furthermore the camera was successfully intercepted and everything should now work properly. Pressing the INS key in the game should now give us a free camera with FoV control. This console will provide you with a help screen but also will state what command is activated. Although IGCS won’t block input by default when you enable the camera, it’s advisable to do so, as moving the camera around with the controller will still pass the controller input also to the game. To block that press numpad-.

Enabling the camera in-game gives us this fancy overview:

We can fly the camera around freely, zoom in/out and look at ourselves, well, at our headless alter-ego at least. Looks good so far, but as you can see we still have some work to do: the HUD is still in view and game pause isn’t implemented yet. Let’s do that now, which will utilize a different technique than we’ve used till now.

Determining hard-coded addresses directly from the exe’s bytes

For the HUD toggle and game pause, we have found hardcoded static variables. Let’s look again how these look. Here’s the g_showhud variable read statement in the game’s exe:

NewColossus_x64vk.exe+F9C173 - 83 3D 669FFF01 00 - cmp dword ptr [NewColossus_x64vk.exe+2F960E0],00

If you pay close attention you see on the right an offset inside the .exe, as source address, NewColossus_x64vk.exe+2F960E0. However if you look at the bytes which actually form this command you can’t find back that offset at all, it’s 0x66FFF01. What’s going on? In x64 assembly, statements often use what’s called ‘RIP relative addressing’. It’s quite simple actually: the offset specified in the command is the offset relative to the instruction pointer’s address (in the RIP register in the CPU) which points to the address the offset is on.

Well, simple for the CPU but for us this isn’t really helpful at all: looking at the offset, we have to calculate what the RIP register will contain at that location, add the offset in memory to it and that will be the actual address the offset refers to. It’s actually more complicated than that, but that’s the gist of it. In Utils.cpp there’s a function, calculateAbsoluteAddress, which accepts an AOB Block with the location of the offset bytes in memory (here, the 0x66FFF01 value) and a value which tells the function where the next statement will be (here 1 byte further as the offset is followed by a 0 byte, so 4 for the offset size and 1 for the ‘00’ byte following the statement) and it will pull the offset from memory, calculate the real address it refers to at runtime and we can then use that in our code.

We now have to find unique byte strings which will find the statements which contain the offsets for the variables we’re after. We’re not going to overwrite any code this time, so it’s ok if there are jmp statements into the blocks. All we need is a unique set of bytes. It’s key to replace the offsets with ?? ?? ?? ?? markers instead, as with a patch these offsets likely change: there’s more/less code in the exe so the memory pages where the variables are on is further/closer to the statement and thus the offset relative to the RIP register changes.

For the g_showhud variable above, we can use the following byte string: 48 8D 54 24 ?? 48 89 44 24 ?? 48 8B CB E8 ?? ?? ?? ?? 83 3D | ?? ?? ?? ?? 00. There’s a ‘|’ character in it too, which tells the AOB scanner to use the address of the byte following it to use as the address to store in the AOB Block instance, not the address of the first byte of the string. We can then use that address as the address of the offset to read from memory and use that to calculate the actual address of the variable, in this case g_showhud. The full statement then becomes:

CameraManipulator::setShowHudAddress(
                     Utils::calculateAbsoluteAddress(aobBlocks[SHOWHUD_CVAR_ADDRESS_INTERCEPT_KEY], 5));

This calculates the address from the address found by the AOB scanner and passes it to the camera manipulator, so when we press the hud toggle key it can set the variable to a 1 or a 0. We’ll use the same mechanism for the other variable we’ve found which offset is hardcoded in the exe: g_stopTime.

Compiling and testing the release build

Now everything’s implemented we can do a final debug run and see if things work. This happens to be the case. However, it’s good to also test a release build as well before shipping the dll out to your friends. The thing is that in C/C++ a debug build gets extra code which makes sure variables are initialized with a default value (usually 0), but in release builds this isn’t the case. Another difference is that release builds are optimized and this could mean the assembler created by the compiler changes a bit which in theory could lead to code that doesn’t work. This isn’t necessarily the fault of the compiler, but often the result of sloppy code in your own project which worked in the debug build because the compiler initialized everything properly but in the release build it crashes.

This is the final console startup output from the release build, with a slightly altered help menu to make things more clear, no debug output here as we’re running the release build, and things seem to work great. We’ve completed our task: creating a photomode for Wolfenstein II: The New Colossus!

Conclusion

In this article I’ve tried to illustrate what steps it usually takes to create camera tools for a game. I hope it taught you something new and was helpful to you too and perhaps it even motivated you to start writing low-level tools like these yourself. Will this now give you the knowledge needed to create camera tools for all other games out there? Not necessarily: every game, even if they use the same engine as another game, might look completely different and it might be you need different tricks to get the information you need. That’s the fun part of this though: you have to outwit the executable with your own set of tricks, come up with new ones. A lot of your attempts might be unsuccessful, but don’t see them as failures: see them as learning opportunities, as you will get better because of them.

The camera tools can be downloaded from GitHub, use that link as well if you want to download / look at the source code.

Thanks for reading and have fun .

Optimizing memory usage

Mon, 04 Dec 2017 16:08:22 GMT

In this post I’ll discuss some memory optimization work I’ve done recently on the LLBLGen Pro runtime framework, v5.3.2. This is a popular (commercial) .NET ORM. LLBLGen Pro is on the market since 2003 and has seen a lot of refactoring work internally over the years, among them performance optimizations and memory usage optimizations. With new features come new challenges to make the overall framework perform the same or even faster and still perform the new features as well, so optimizing the core engine is a recurring effort.

The optimizations discussed here form just a small subset of the massive amount of things you can do to optimize memory usage, but it could give some insights in how to get started and what to look for in your own code. As with all optimization work, it depends on the situation at hand and why that particular piece of code is less optimal so you can fix it properly.

Before we can do any optimization however, we first have to define a couple of things so we don’t do any work unnecessary or even harm the code we want to make more optimal. I won’t go into fine details about the differences between the various heaps, as that’s besides the scope of the article.

A word of warning though: pre-mature optimization is bad. The things discussed here are solutions to problems I recognized. They weren’t applied because ‘they are faster, regardless’; they’re not rules you should live by. Sure, if you have the choice between two equal solutions, just pick the one which is more efficient. Performance optimization isn’t free: there’s always another thing you can optimize, and all it takes is a compromise elsewhere. Is that price worth paying? Up front you don’t know that, hence optimizing right from the start isn’t going to be beneficial in many cases: write clean, simple code that’s easy to maintain and understand. If there’s a slightly faster variant thinkable that’s way more complex and harder to understand, chances are it’s not worth it, and you only know after the fact, never before the fact: the code might be called just once, so optimizing it away is useless.

Why optimize memory usage and when?

.NET is a garbage-collector (GC) based platform and that brings great joys because you don’t have to care about memory allocation and management: the GC cleans up after you’ve trashed the room and in theory you never have to worry about any consequences. While trashing rooms as a wanna-be rock star might feel exhilarating, there’s no free lunch: not in hotels, nor inside your computer beneath the .NET CLR and its digital vacuum cleaner; something has to pick up the tab, and in the context of this article, it’s the GC.

Allocating memory has a cost, albeit in general a small one: it takes a bit of time to allocate some memory, initialize it and make it look like that fancy StringBuilder you just new-ed up. When you add some strings to that StringBuilder and its internal buffer is too small, it has to allocate a new block of memory, copy over the original contents and free the old block. Those actions take time. Not ‘copy this large file to a usb-stick’-time, but doing it a lot of times will be noticeable.

Besides the work the GC has to do for cleaning up the old buffer, the whole allocating/copying/freeing dance has another side effect: memory fragmentation. The GC/CLR compacts memory when it gets the chance. The main reason is that if your program needs to have a large block of continuous memory, chances are it’s available, and not fragmented over a million already deallocated buffers. In the latter case you’d get an out-of-memory exception as the block you’re requesting isn’t there.

So in short: all that memory allocating work can be a serious burden for the overall performance of your software. The less memory your application allocates, the better: what isn’t allocated doesn’t need to be cleaned up / copied / packed and taken care of. The problem of course is: with a system where memory management is taken care of behind your back, you’re not focused on it when writing code and reading a piece of code won’t immediately reveal where things are less optimal.

Profile, Analyze, Fix

You can’t say anything about whether software is slow or less optimal unless you’ve measured it first. This is done using a profiler. A profiler is a tool which measures performance, memory usage and the like of software and gives you fancy reports about what code was executed, how slow each statement was and what memory was used. There are several profilers for .NET, e.g. Jetbrains’ dotMemory and dotTrace, Red Gate’s Ants and e.g. the visual studio performance analyzer. Each has their strong points and weaknesses, and some offer some features others lack, it really depends on what you’re after. For this work I’ve used the visual studio performance analyzer, but in general what matters is: at least use a profiler. Make it a part of your every day tool-chain. If you’ve never used a profiler before, make it your main task tomorrow and run one over your code. Just to see what it digs up. Changes are you’ll be surprised what’s really going on inside your software.

With the profiler ready to be used, you have to define a set of tests which you want to measure. For this article I’ll be looking at some fetch benchmarks I’ve performed recently, which you can see in my previous post “.NET (Micro)ORM fetch benchmark results and the fine details”. It matters to have a context for your tests: if your measured code does things in N seconds and consumes M megabytes of memory, how can you say that’s an OK result or a very poor result? If you have comparable systems / pieces of code available, these can give context: in the light of my benchmarks: the hand-written, hand-optimized code is a good example how optimal things can be.

When examining the results a profiler gives you, you have to compare results with what’s realistically optimal. Take the hand-optimized benchmark: it’s very fast, but offers no other features as well. For instance there’s no way that code does authorization. If your code is slower or consumes more memory than that fastest variant, is that because you simply have more features and those take some performance, or is it because you made some sloppy mistakes and can do better? In short: the code you wrote, is it acceptable it’s that slow and eats that much memory? If comparable systems all do, maybe even more than your code, it might very well be your code is near optimal for what it does: you might optimize this code just for fun, but it’s likely you won’t find much to work with.

After you’ve found areas which are less optimal, you have to analyze what the real problem is: do you do unnecessary work, do you allocate a lot of objects you don’t need, or e.g. did you use code that was slammed together without any thought about performance and memory usage? Just three examples out of many which all need their own unique approach how to fix them. It might also be there’s no real fix for the code you have or it would require a complete rewrite of a lot of subsystems. That’s OK: at least you measured it, you analyzed it, and you made an effort to make it better but the costs are too high right now.

When to optimize?

People who just start using a profiler will often try to optimize everything, but that’s a mistake. Doing things with a computer takes time, it does use memory, it will not be ready in an instant. In general I use the rule that optimizing everything that consumes less than 10% of the overall time or less than 10% of the overall memory used by the test is likely a waste of time: even if you optimize it completely away, you gain at most 10%. Chances are if you are lucky, you’ll get from 10% to 7% or maybe even 5%, so you gained 5%. That’s not a big win. Instead first look at areas which take 40%-50% or even more and see what takes all that.

Visual Studio Memory Profiler

For the particular work we’ll be doing here, it’s key to have the right tooling. I’ve used the visual studio .NET memory allocation profiler for that. It offers a call chain with per method the bytes allocated. Most other profilers don’t offer this particular view, as they’re mainly focused on what memory is currently allocated at time T and what code allocated those objects, but it’s crucial for the measurement of which code is allocation more memory than it should. To learn more about this profiler, please see this excellent article from Stephen Taub: .NET Memory Allocation Profiling with Visual Studio 2012. Although it’s a few years old, it’s applicable to the current visual studio versions as well. For this work I’ve used the Call graph view and included the percentage columns (not visible by default) so you get a good overview what to ignore and what to focus on.

With that knowledge under our belt, let’s check out some optimizations I did and why.

Optimizing the Linq provider

If you look at the ‘Individual fetches, non-change tracking’ graph in the benchmark article, you’ll see that the Linq provider of LLBLGen Pro does consume more memory than all the others (the red line). Processing Linq expression trees is a complex affair, and doing things in the most optimal way is often not the first concern when writing a full Linq provider. Still, doing things inefficiently is of course something we should avoid.

Using the .NET memory allocation profiling feature of the performance analyzer revealed a couple of things. I’ve described them below.

Avoid params arrays

The Linq provider calls in every method at the start and end a method which traces the method start and method end. This method ends up in this simple tracer helper method:

   1: public static void WriteLineIf(bool condition, string messageInFormat, string category, params object[] argumentsForMessage)   2: {   3:     if(condition)   4:     {   5:         Trace.WriteLine(string.Format(messageInFormat, argumentsForMessage), category);   6:     }   7: }

The problem here is that with every call the argumentsForMessage array is allocated (and destroyed right after the call by the GC). Calling this method many times (and in a Linq provider, the visitors’ methods are called a lot) will make the CLR allocate a lot of object arrays which are then destroyed right after, causing potentially memory fragmentation. We can avoid these by simply making sure we hit an overload which doesn’t allocate the object arrays but accepts a fixed number of arguments. For this particular hot path this was ideal, as the method start / end trace messages were effectively simple strings and didn’t need string formatting.

Avoid StringBuilder.AppendFormat()

Inside the Linq provider it’s sometimes necessary to define a ‘key’ for a subtree or expression: You can then e.g. assign an alias on that subtree or expression. This is important as expression trees get transformed into different trees along the way, and when doing so you lose the object references you had as the expressions in an expression tree are immutable. So I calculate keys based on strings: I use visitors traversing the sub tree which along the way build a string from what they encounter, based on types and values. I use code like this:

   1: internal static void AppendExpressionFragment(StringBuilder toAppendTo, LinqExpression expression)   2: {   3:     if(expression == null)   4:     {   5:         return;   6:     }   7:     toAppendTo.AppendFormat("NT{0}|T{1}|", (int)expression.NodeType, expression.Type.GetHashCode());   8: }

This uses an AppendFormat, but it’s actually unnecessary. More efficient is the following:

   1: internal static void AppendExpressionFragment(StringBuilder toAppendTo, LinqExpression expression)   2: {   3:     if(expression == null)   4:     {   5:         return;   6:     }   7:     toAppendTo.Append("NT");   8:     toAppendTo.Append((int)expression.NodeType);   9:     toAppendTo.Append("|T");  10:     toAppendTo.Append(expression.Type.GetHashCode());  11:     toAppendTo.Append("|");  12: }

This is a simple change, but has a big benefit. All allocations needed for the format handling are avoided.

Pre-size data-structures if you add data immediately afterwards

Consider the following visitor handler:

   1: protected virtual IEnumerable<ElementInit> HandleElementInitializerList(ReadOnlyCollection<ElementInit> listToHandle)   2: {   3:     TraceScopeStart("GenericExpressionHandler::ElementInitializersList");   4:     List<ElementInit> newList = new List<ElementInit>();   5:     bool changedElementInitializerSeen = false;   6:     for(int i = 0; i < listToHandle.Count; i++)   7:     {   8:         ElementInit currentInitializer = HandleElementInitializer(listToHandle[i]);   9:         newList.Add(currentInitializer);  10:         changedElementInitializerSeen |= (currentInitializer != listToHandle[i]);  11:     }  12:     TraceScopeEnd("GenericExpressionHandler::ElementInitializersList");  13:     return changedElementInitializerSeen ? (IEnumerable<ElementInit>)newList : listToHandle;  14: }

A new List<T> is initialized with length 0. This means it doesn’t allocate any memory for its buffer. If you add a single element, it has to allocate a buffer. But it doesn’t allocate a large buffer initially. When adding the initial element, it will allocate an array of 4 items. If you add 5 elements, it thus has to throw away the array of 4 items, allocate one which can contain at least 5 (it uses a scale for that, see the List<T> source code) and copy over the 4 items already in the old array.

In the method above, we already know how many items we’ll add, namely listToHandle.Count. We can initialize the List<T> at line 4 with this number and make sure List<T> won’t re-allocate memory during the for loop. The careful reader can spot another potential optimization that’s not implemented here, as it would make the code more complex, do you see it (hint it relates to the next topic) ?

Initialize members lazily

In a Linq provider you track a lot of things along the way, e.g. which elements were aliased, which scopes are currently active and many more things. These are all stored in datastructures like dictionaries, lists and stacks. I use a class for that called MappingTracker, which tracks the collected data in its internal structures. Every visitor receives the general MappingTracker instance or if it’s a temporary visitor, a new instance. Being a good sport, I simply created new instances of these internal structures in the constructor, to avoid null checks and creations all over the place. However the price for this turned out to be higher than expected: every time the Linq provider creates an expression key (as shown above) it created a new tracker and a new set of these internal structures.

To fix this, simply initialize the structures lazily. I’ll give two examples. The first is the way which allocates memory for the datastructure regardless if it’s used or not. The second initializes it lazily.

Old:

   1: public class MappingTracker   2: {   3:     private readonly HashSet<SetAlias> _createdAliases;   4:     //...   5:        6:     public MappingTracker()   7:     {   8:         _createdAliases = new HashSet<SetAlias>();   9:         //...  10:     }  11:       12:     public SetAlias CreateNewAlias()  13:     {  14:         _aliasCounter++;  15:         SetAlias toReturn = new SetAlias("LPLA_" + _aliasCounter);  16:         _createdAliases.Add(toReturn);  17:         return toReturn;  18:     }  19:    20:     //...  21:     

New:

   1: public class MappingTracker   2: {   3:     private HashSet<SetAlias> _createdAliases;   4:     //...   5:        6:     public MappingTracker()   7:     {   8:     }   9:       10:     public SetAlias CreateNewAlias()  11:     {  12:         _aliasCounter++;  13:         SetAlias toReturn = new SetAlias("LPLA_" + _aliasCounter);  14:         this.CreatedAliases.Add(toReturn);  15:         return toReturn;  16:     }  17:    18:     //...  19:    20:     private HashSet<SetAlias> CreatedAliases  21:     {  22:         get { return _createdAliases ?? (_createdAliases = new HashSet<SetAlias>()); }  23:     }

The access to the member is now through a private property which allocates when needed (only the first time). If no call is made to this particular method, no HashSet<SetAlias> is created. With a good refactoring tool like Resharper, this change is easy and painless.

Avoid Linq-to-Objects if necessary

It turns out using .Any() or .FirstOrDefault() on e.g. a HashSet<T> isn’t free from allocations. If this method is on a hot path as well, chances are the allocations are seriously noticeable. Consider the following code:

   1: HashSet<SetAlias> aliases=new HashSet<SetAlias>();   2: if(!string.IsNullOrEmpty(expressionKey))   3: {   4:     SetAlias keyAlias = null;   5:     if(_aliasPerExpressionKey.TryGetValue(expressionKey, out keyAlias))   6:     {   7:         aliases.Add(keyAlias);   8:     }   9: }  10: if(!aliases.Any())  11: {  12:     if(!_aliasesPerMemberInfo.TryGetValue(info, out aliases))  13:     {  14:         aliases = new HashSet<SetAlias>();  15:     }  16: }  17: toReturn = aliases.FirstOrDefault(a => CheckIfAliasIsInScope(a));

Refactoring it to the following solves allocations and is faster. As it’s a hot path method I went for this refactoring, as it paid off, but in general you might think this is a bridge too far because it’s more code and potentially more complex.

   1: HashSet<SetAlias> aliases = null;   2: if(!string.IsNullOrEmpty(expressionKey))   3: {   4:     SetAlias keyAlias = null;   5:     if(this.AliasPerExpressionKey.TryGetValue(expressionKey, out keyAlias))   6:     {   7:         aliases = new HashSet<SetAlias> { keyAlias };   8:     }   9: }  10: if(aliases==null || aliases.Count <=0)  11: {  12:     this.AliasesPerMemberInfo.TryGetValue(info, out aliases);  13: }  14: if(aliases != null)  15: {  16:     toReturn = null;  17:     foreach(var alias in aliases)  18:     {  19:         if(CheckIfAliasIsInScope(alias))  20:         {  21:             toReturn = alias;  22:             break;  23:         }  24:     }  25: }  26: if(aliases!= null && toReturn==null)

When arriving here, I cut down the 550KB per query to 320KB or so. Still not the result I’d have hoped, but it was a lot less than where I started with.

There was more to find of course, namely in the SQL generator.

Optimizing the SQL generator

In an ORM, generating SQL strings is one of things it is meant to do, and if you’re not careful it will generate a lot of strings. Not only the final SQL query string, but also name fragments, concatenated parameter names and other things. I’ve optimized this system over the years as it’s one of the main time sinks of a query pipeline and a major memory waster. Some time ago I refactored the query pipeline to use a list of fragments, which of themselves could be list of fragments. This was abstracted away into a QueryFragments class and some minor classes for e.g. a delimited list of fragments. Internally these classes simply collected fragments like a name of a field, and by building these fragment structures during the SQL generation you could build the query decentralized and without concatenating strings all over the place, that was simply done at the end: by flattening everything to a string.

This looks like this

   1: var fragments = new QueryFragments();   2: fragments.AddFragment("SELECT");   3: StringPlaceHolder distinctPlaceholder = fragments.AddPlaceHolder();   4: StringPlaceHolder topPlaceholder = fragments.AddPlaceHolder();   5: QueryFragments projection = fragments.AddCommaDelimitedQueryFragments(false, selectList?.Length ?? 0);   6: QueryFragments fromClause = fragments.AddQueryFragments();

After this code, methods then could add fragments to the ‘fromClause’ or ‘projection’ without worrying it would insert strings into another string, neither does the code have to be in a given order to construct the string, it can be in any order: the code just has to make sure it adds its fragments to the right fragment container.

You can’t always avoid creating temporary strings along the way. For these situations I used StringBuilders with an estimated size. In v5.3 I optimized this further with a StringBuilderCache, which was a borrowed class from CoreFX (from System.IO). The idea is simple: you ask the cache for its cached StringBuilder, which already has a pre-allocated size. If it’s available you return it and un-cache it (it has just 1 cached instance). If it’s not available you simply allocate a new one. When you’re done you add it back to the cache. If there’s already one cached but it’s smaller, you toss the old one away and cache the one with the bigger internal buffer.

Profiling this code using the memory profiler I still saw a lot of StringBuilders being created and more importantly: being resized. So I rewrote the StringBuilderCache to keep 4 around instead of 1. The code can be found in this gist. If now during the flattening of a hierarchy of QueryFragments multiple StringBuilder instances are at play, it will avoid reallocation of buffers and allocating more StringBuilders. So how to use this cache? The below snippet shows how:

   1: StringBuilder sb = StringBuilderCache.Acquire(originalSqlString.Length + this.QueryTag.Length + 8);   2: sb.Append("/* ");   3: sb.Append(this.QueryTag);   4: sb.Append(" */ ");   5: sb.Append(originalSqlString);   6: this.Command.CommandText = StringBuilderCache.GetStringAndRelease(sb);

Here it first acquires a new StringBuilder, or if one is already present with the requested size, it gets one from the cache. It then appends some strings, and on the last line it gets the string from the string builder and releases it back to the cache.

Using multiple instances in the cache also means in recursive scenario’s like in this situation it will still have potentially a cached builder around. For my situation 4 turned out to be more than enough.

End results

After a week working on this, I got the single element fetches which use Linq down from 1.54ms to 0.95ms and from 553KB per element to 280KB per element. I think that’s a lot of progress Additionally, the optimizations in the SQL engine also benefited the LLBLGen Pro QuerySpec query pipeline (QuerySpec is LLBLGen Pro’s fluent DSL for building queries in C#/VB.NET), where memory usage went down from 121KB to 80KB per element and performance went from 0.65ms to 0.59ms.

All in all very happy with the results of a relative small amount of changes. These changes are available in LLBLGen Pro v5.3.2, currently available as hotfix build.

Happy optimizing!

.NET (Micro)ORM fetch benchmark results and the fine details

Mon, 20 Nov 2017 15:41:30 GMT

For some time now I maintain the RawDataAccessBencher repository on github, and once in a while I run the suite of benchmarks and post the results. In the benchmarks I included all major micro-ORM and full ORM frameworks for .NET which have a significant group of users. The benchmarks are typical ‘micro’ benchmarks in that they run for a relatively short period of time. They also focus solely on fetch performance, how fast the given framework can create a query, fetch the resultset from the database server (on another machine on the network) and materialize objects from the resultset.

In this article I’d like to present the results of the run of today. You can look at the results in a flat list here, or jump straight to the conclusions, by looking at the final results here. So how to read those values? The main point of this benchmark is to illustrate how fast a given framework is compared to its competition for set fetches and individual fetches. The numbers, while interesting, can’t be extrapolated to your own system. So if in the list of results framework X takes 300ms to fetch a set of rows, it doesn’t mean it will take 300ms on your hardware. It will only show it’s faster than framework Y which takes 400ms for the same data. It’s also likely X will be faster than Y on your system or any other system.

I’ve updated RawDataAccessBencher recently with memory allocation values as well, so it’s now also more clear how much memory a framework consumes during a benchmark run. As some of you have asked me about graphs of these results, I’ve put the values in Excel and created some graphs for you to give a more visual overview who’s fast and who’s dropping the ball. I’ve combined the memory consumption graph with the timing graph so you can quickly see if a fast performance comes with a high price (lots of memory) or not.

As data source, the Sales.SalesOrderHeader table is used from AdventureWorks. This is for a reason: it has fields with different types, has a couple of relationships with other entities, relationships of different kinds, has nullable fields and has a reasonable amount of data (31K rows). The benchmarks are divided in two groups: change-tracking fetches and non-change-tracking fetches. This is done mainly to avoid an apple-oranges comparison, as change-tracked fetches can take more performance than read-only, non-change tracked fetches.

If you’re interested, please run the benchmark code on your own system. All dependencies are obtainable from NuGet, you only need the right database data/schema available in a SQL Server database, change the connection string and you’re good to go.

The graphs

With the graphs below, I’ve tried to explained why they are the way they are. All benchmarks were run on .NET 4.7, x64, release builds, clean network. Even though my own work is included in the benchmarks, I’ve tried to keep things fair.

For all graphs: lower is better, the fastest framework / variant is placed at the left. Some frameworks have multiple results in one benchmark, e.g. because they offer multiple APIs or offer ways to pre-compile queries, which affect results. Each graph therefore is more than just the winner and the loser, ok? Good

Ok, let’s dive in!

Set fetches, non-change tracking

This is a straight-forward fetch: fetch all rows in the table Sales.SalesOrderHeader into objects and don’t do any change tracking.

Here, the hand-written, hand-optimized materializer using a DbDataReader is the fastest. This is no surprise, there’s little room here to get any faster. On the other side of the spectrum we’re seeing Massive, which uses Expando/Dynamic to materialize the rows. I happen to be the maintainer of Massive and of course have profiled the codebase to see where the performance is lost. This came down to the caching mechanism implemented in the Expando implementation inside .NET. The only way to make this much faster was to implement my own Expando class. As these then wouldn’t be interchangeable with vanilla .NET Expando instances, I haven’t made this change.

The vast majority of the results are in a narrow range, from PetaPoco Fast till LLBLGen Pro, poco with Raw SQL: ranging from 161ms till 168ms. I think the main reason for this is that they’re all using the same trick, more or less: compile a piece of code at runtime, which receives a DbDataReader and which returns an instantiated object with the data of the current row. You can construct this code in various ways, from generating IL at runtime till constructing a lambda and compile it using its Compile() method. You then cache this compiled variant and re-use it with every row. There’s one framework which also does this, but is still faster than this group: Linq to DB. Profiling this framework it shows it not only constructed a lambda (compiled) which then materializes the object, but it was able to guess which fields would likely be NULL or not.

As you can see in the handwritten materializer code, not all fields can be NULL. Most frameworks are testing each field if it’s NULL, and if not, obtain the value, otherwise either ignore the column or set it to null. The reason Linq to DB is faster than the rest is that it skips testing for NULL for fields which can’t be NULL. It does this by examining the schema returned by the DbDataReader. This is risky however. It doesn’t always match with reality. When I discovered how they did this, I started implementing this in the LLBLGen Pro pipeline, but I ran into some problems and I remembered again why: this meta-data isn’t always up to date. Especially in SQL Server, with Views, it can be seriously outdated: if a table’s meta-data is updated it won’t update the meta-data for a view (so it will use the outdated table meta-data) which depends on that table. It will only do so when you drop and recreate the view. Other databases can do this as well. It would be interesting to check whether this is the reason why Entity Framework Core doesn’t support views (yet).

Looking at the memory usage, it’s interesting to see that the framework Microsoft pushed for cloud-apps, its own Entity Framework Core, uses more memory than most other frameworks, and isn’t faster either, on the contrary. In that light, I’m pleased with the results of LLBLGen Pro’s raw SQL API: high performance and low memory usage.

Set fetches, change tracking

This is a straight-forward fetch of entity instances which are change-tracked. This means you can change the value of a field of such an entity object and persist it again to the database using the ORM framework / data-access layer. Tracking changes can have consequences for fetch code, and it’s no surprise the results of change tracked fetches are a bit slower than the non-change tracked results. There’s no free lunch.

It’s quite obvious there are two frameworks which have a serious performance problem here: NHibernate and Entity Framework 6. They’re not only seriously slow, but also consume a massive amount of memory to materialize the entities. Profiling NHibernate v5’s fetch code reveals it’s losing performance all over the place with very long call chains (EF6 has this too btw) in very complex code, and what’s also interesting is that it pays a serious price for its Poco philosophy: the proxy classes it has to create for related elements (EF6 has the same problem) really bog it down. Entity Framework Core doesn’t have this problem, yet, as it doesn’t implement lazy loading, so it can get away with avoiding introducing proxies or at least inject code which will trigger lazy loading of related elements.

Again, I’m happy with the results my own work shows here: LLBLGen Pro’s entity fetch pipeline is very efficient, both in memory consumption and performance. It uses a slightly different philosophy to change tracking: all change tracking is done by an entity itself, so no complex bookkeeping or manual graph handling when you want to save some entities; they know what’s changed already, no need to compare their state with some in-memory state of a shadow entity in some context you need to keep alive.

What’s every time a shining light of how Microsoft mismanaged its own data-access APIs is Linq to SQL. It keeps kicking Entity Framework (core)’s ass, while it’s barely updated nowadays. Oh and don’t forget the DataTable based approach, which is hard to beat, even in 2017. I doubt we’ll ever beat it. I for one don’t know any more tricks to add to my own framework without cutting corners to get passed it and I doubt others will too.

Individual Fetches, non-change tracking

This is a fetch of individual rows, based on a given PK. It will fetch 100 rows based on a previously fetched set of 100 PK values, one row at a time, utilizing the complete fetch pipeline, so not a tight loop. This fetch gives a high emphasis on query construction even though the query is fairly simple, compared to pure object materialization in the set fetches. This benchmark is for non-change tracking fetches, so all frameworks fetch read-only non-change tracked objects.

Here we’re seeing some interesting results. First, the hand-written, hand-optimized materializer is surpassed (with a fraction) by a couple of frameworks, illustrating how close together this group is (with a delta of 0.15ms). Then, the slowest non-change tracking framework in the set benchmark, Massive, is now one of the fastest. The main reason is again Expando. The call chain inside Massive is very short, and as it’s just one row, the part which makes Massive slow in set fetches, the Expando caching system, is never invoked.

Slowest is PetaPoco (though we’re talking 3.37ms), which seems to be caused by the fact it’s not really optimized for this scenario; it seems it doesn’t cache the projection code I mentioned earlier, so for every query it has to rebuild that. Second slowest is LLBLGen Pro using its Linq API. If I compare this with the results of LLBLGen Pro using its QuerySpec API and the results using the LLBLGen Pro raw SQL API (3rd fastest), it’s clear the Linq pipeline is the culprit here: both the Linq API and the QuerySpec API translate to the Low-level API (not in the benchmark) and its objects, which translate to SQL. The materialization pipeline is roughly the same as the raw SQL API materialization with one difference: it has to take into account type conversions. Per query it has to do this work, once, hence the difference with the raw SQL API. The more work one has to do per query, the slower it will appear in this benchmark.

The Linq variant also consumes more memory than the QuerySpec variant, mainly because the framework doesn’t use any expression tree caching. This means it has to transform the whole expression tree to SQL every time, which can be costly memory wise, as every expression transformation requires a rebuild of the tree using newly created objects (as expression tree elements are immutable). It’s a choice whether to cache the expressions coming in or evaluate it every time. Caching the trees can be efficient in scenarios where a small set of queries is used in a high frequency. If queries differ a lot, caching the trees isn’t really beneficial. It’s however difficult to decide when the tipping point occurs. For now I’ve chosen not to implement this kind of tree caching, as LLBLGen Pro offers 4 different query systems, it’s often easier to optimize individual queries using one of the other query systems than to implement a caching system which is beneficial in only a a subset of the cases and which hurts in a lot of other cases.

Individual fetches, change tracking

This is a fetch of individual rows, based on a given PK, it’s the same as with the non-change tracking individual fetches, except this time, change tracked entities are fetched, one at a time.

The same thing we saw with the non-change tracked individual benchmarks is also present here: the slowest of the set fetches is now one of the fastest. Here too we see a Linq based framework struggling for memory consumption, this time it’s Linq to SQL. Although Entity Framework Core uses a Linq query (a Single() call) as well, I suspect they’re using query caching along the way, as rewriting the query from a Single(lambda) to a full from select with a FirstOrDefault call gave the same results; the latter tree much deeper than the former, which suggests a caching of expression tree –> SQL conversions.

Eager loading, 3-node split graph

This benchmark fetches a split, 3-node graph of change-tracked entity instances using eager loading: one ORM query fetches the complete graph. It fetches 1000 SalesOrderHeader entities and their related SalesOrderDetail entities and Customer entities.

It’s likely not a surprise that the performance of set fetches in an eager loading scenario is the same as with the set fetches earlier: NHibernate and Entity Framework 6 are embarrassingly slow and especially in the case of NHibernate consume a lot of memory. What’s interesting is that there are different scenarios used by the frameworks: as it’s a split graph, it might be better to choose for one query per graph node, instead of joining everything together (to avoid duplicate rows). Linq to SQL still comes up top, even though it does use the Join approach. LLBLGen Pro uses a one query per node approach, same for Entity Framework Core. The number of root entities is set to 1000 which is outside the window where the LLBLGen Pro query optimizer will optimize the eager loading queries by switching from sub queries to parameters.

Eager loading, 3-node split graph, async

This benchmark is equal to the previous one, however this time an async query is used. The benchmark doesn’t use async queries except in this benchmark as it makes more sense to test it in this scenario with multiple database queries issued by the ORM.

The usual suspects are on the right side of the graph, but what’s interesting is Entity Framework Core compared to LLBLGen Pro. Not only is Entity Framework Core’s performance significantly slower than in the synchronous variant memory usage is also up, where LLBLGen Pro’s sync performance is very close to the async performance and memory usage is roughly the same. This puzzled me a bit the first time I saw it. At first I was afraid I made a serious mistake somewhere and the async code in LLBLGen Pro accidentally ran as synchronous but profiling the code again clearly revealed what was expected: it was async from start to finish.

What could explain the difference? I have to guess, but I think it might be call chain length combined with the nasty aspect of async code: fragmentation of code bases. With async code, it ripples through your code like wildfire: every method that’s async has to be called by an async method. If that code is also shared with the synchronous API, you can do two things: wrap it so the async code runs synchronous in the synchronous variant (which likely will cause a performance hit) or factor out the code which is shared.

This factoring out might cause performance problems for async or for sync paths. A long call chain causes lots of statemachines being created and this too could lead to less optimal performance. All in all, it’s not an easy fix.

Conclusion

What conclusion can one draw from the information provided? If I may, I’d like to advice you to look at all the graphs. Your application is likely not only doing read-only fetches, but will also fetch single entities, large sets, will modify sets in memory, will fetch graphs. It’s good to see how a given framework performs over all the graphs provided. Also keep in mind this is just a snapshot of what an ORM has to offer.

For modern apps, which more and more move to the cloud, memory consumption as well as high-performance are key to keep costs low. It’s therefore important to look for a complete framework which offers low memory consumption, high performance and allows you to fetch the data the way you want: with high-level queries when you want to, and low-level SQL when you need to.

LLBLGen Pro v5.3 released!

Mon, 16 Oct 2017 07:48:55 GMT

Despite some large companies deciding they can go from preview to RTM in a week, I still think following the steps alpha-beta-RTM is better, so after the beta period of LLBLGen Pro v5.3, there’s now the RTM version of v5.3! For what’s new, you can read that in the beta announcement: https://weblogs.asp.net/fbouma/llblgen-pro-v5-3-beta-has-been-released.

The version is free for everyone with a valid subscription.

Enjoy!

LLBLGen Pro v5.3 Beta has been released!

Thu, 28 Sep 2017 14:49:47 GMT

We've released LLBLGen Pro v5.3 beta! Since the EAP we’ve added new functionality and tweaked some things too, based on feedback. Below is the full list of what’s new in v5.3 Beta, and this is also the list of new stuff we’ll include in v5.3 RTM, which is expected within a week or two.

What's new in the LLBLGen Pro designer

.NET Standard 2.0 support. The designer can now generate, for selected target platforms, .NET Standard 2.0 source code
Firebird 3.0 support. LLBLGen Pro now supports Firebird 3.0, in both designer and LLBLGen Pro runtime. This effectively means that the new Boolean type is now fully recognized. The identity field support of Firebird 3.0 will be added as soon as the Firebird Client supports this.
The latest releases RSS feed contents is now shown on the Hometab. This way you know when a new release is posted on the website.
Default sequences for Postgresql in model first. When using model first with PostgreSql, the designer will for fields which have a sequence set in their field mappings set the sequence as the default sequence in the meta-data and will generate the sequence as the default value in DDL SQL scripts. This is needed for ORMs like Entity Framework 6 which don't have sequence support in their mappings and rely on default values for identity/sequence support.
New Project setting: StoreTimeLastSyncIntoProject. A new project setting (under Entity model - General) has been added: StoreTimeLastSyncIntoProject which, when true (default) will make the designer store the last sync time with database or model with the catalog / schema. This setting can be switched off to e.g. avoid projects having merge conflicts based on time/date values of last-sync

What's new in the LLBLGen Pro Runtime Framework

New functionality / changes

.NET Standard 2.0 support. The LLBLGen Pro Runtime Framework can now be used on .NET Standard 2.0 supporting platforms like .NET Core 2.0 and Xamarin.
More information
New RuntimeConfiguration system to configure LLBLGen Pro Runtime Framework specific behavior and settings in code instead of in .config files. Mainly meant for .NET Standard 2.0 using applications but it's also available for .NET Full.
More information
The Plain SQL API now has a FetchScalarQuery<T> method to retrieve a scalar value from the DB using a plain SQL query.
More information
Query tagging. Linq and QuerySpec have the ability now to let you mark (tag) the SQL query or queries generated by the runtime with a tag. This tag is a string which is added as a comment before the actual query string. This way you can easily track back the SQL query on a database server with the actual calling code without a profiler / stacktrace. It's a low-impact way to make sure you can track down information at runtime in the future.
More information: Linq, QuerySpec.

Small changes

The Runtime Framework is now compiled against .NET 4.5.2
QuerySpec: When a scalar query is executed using FetchScalar<T> and T isn't a nullable type and the query returns a NULL value, the cast to T would result in a Null reference exception. We now added a check first which will throw an InvalidOperationException if a null value was returned and the type specified isn't a nullable type.
Lambda caches and factory caches are now using a ConcurrentDictionary instead of a Dictionary per thread, which should save some memory in large applications.
QuerySpec: an extension method has been added for ISortClause, DontEmitAliasForExpressionAggregateField, so the EmitAliasForExpressionAggregateField flag can be set using the fluent interface. See the LLBLGen Pro Runtime Framework Reference Manual for details.
A new exception class has been added: ORMConfigurationException. This exception is thrown when an error is detected in the RuntimeConfiguration usage.
Prefetch path fetches with a context are now using faster merges as the linear searches caused by the context usage in collections during merges are now no longer happening. This is done when the fetch is into a new graph. An existing graph will do the linear searches during merge as before.
Nuget: the packages for DQEs now reference a specific ORMSupportClasses version, which means you always have to update both when referencing a specific DQE package version. Previously we defined a range of [currentversion-nextmajorversion) but that is too large and causes 'downgrades' for .NET Core due to a change in the way downgrade warnings are handled by the .NET Core tooling.
Field compare value predicate now detects value truncation for variable string typed parameters. If it detects truncation, it will emit "1=0" instead of the actual predicate, followed by a comment to make the predicate result in 0 rows. Value truncation occurs if the DbParameter's Value property is set to a value which has a length longer than the 'size' property. The size property of the parameter is set to the length of the field compared against. So if you compare a varchar(20) typed field F to a string with length 25, you'll get 1=0 instead of the comparison of the first 20 chars of the string with the field, which could lead to matches which are unintentional.
Field value equality comparisons of byte array values are now more optimal.
Firebird 3.0 support. LLBLGen Pro Runtime Framework now supports Firebird 3.0. This effectively means that the new Boolean type is now fully recognized. The identity field support of Firebird 3.0 will be added as soon as the Firebird Client supports this.

What's new in the Entity Framework support

The generated POCO classes for CodeFirst / DbContext as well as for EF Core 1 and 2 now have a partial method OnCreated() which is called at the end of the ctor. This can e.g. be used for custom types for navigators to be created automatically, e.g. when a specific sorted list is needed for a specific navigator.
Entity Framework Core v2.0 support. EF Core 2.0 is supported on .NET 4.6.1 or higher and .NET Standard 2.0.
Entity Framework Core 1.0 is now marked 'legacy' in the designer. New projects should use Entity Framework Core v2.0.

What's new in the NHibernate support

It's now possible to specify whether a normal field in an entity should be lazy loaded, using the field setting LazyLoadField. Default is false.

All our runtime framework dlls are on nuget as well, but have a prerelease suffix, so to see them you need to enable 'prerelease' in the nuget client in visual studio.

To download the v5.3 Beta you have to have a valid subscription. Go to: https://www.llblgen.com/ -> My Account -> V5.3 and then the Betas section

LLBLGen Pro v5.3 EAP Released!

Thu, 07 Sep 2017 08:56:01 GMT

After a failed attempt to port our LLBLGen Pro Runtime Framework to .NET Standard 1.6, we’ve ported it to .NET Standard 2.0 and managed to port over almost all features (A very small set of features aren’t supported but you either won’t miss them, or they’re out of our hands and limitations of .NET Core/standard)! This means finally there’s a full O/R mapper framework available for .NET Core 2.0 that’s fast, fully featured and ‘battle-tested’ for many years.

The port of our runtime framework comes with LLBLGen Pro v5.3, which we’ve released an Early Access Preview (EAP)version of yesterday. What’s also in v5.3? Well, Entity Framework Core 2.0 support. The most notable additional features for EF Core 2.0 support from our perspective are entity splitting, valuetype definition support and if you are into it, VB.NET

We’ve also changed/added some small things in the designer and runtime but these are relatively minor. We’ll be adding more features before RTM, which we expect to happen within a month or so.

I know a lot of people have been eagerly waiting .NET Core support for our runtime and I’m pleased to say it’s finally there. As the port is almost 100% feature compatible with the version for .NET Full, porting your own application to .NET Core should be relatively easy, or better: the ORM at least isn’t the problem here.

Stay tuned for more!

LLBLGen Pro v5.2 has been released! (and first quickstart video posted)

Thu, 04 May 2017 12:16:55 GMT

LLBLGen Pro v5.2 has been released! It’s a version with a couple of ‘highlight features’ like full model analysis and a lot of smaller new features / changes which together form a large list of new goodies, see my previous post on LLBLGen Pro v5.2.

We also started creating some videos, which are meant to get you started with LLBLGen Pro and show you how to get results very quickly, using various tools in the designer and the different ways people want to work (Model first, Database First). The first video is now done and it’s embedded below. It’s about getting from just a database to working sourcecode using database-first modeling. The next video will show how to get from scratch to a working database and working code using model-first modeling.

Enjoy!

From Database To Code, using LLBLGen Pro

LLBLGen Pro v5.2 Beta has been released!

Fri, 07 Apr 2017 09:03:10 GMT

Earlier this week we've released LLBLGen Pro v5.2 beta! See below for the new features and changes in this release. It's feature complete and comes with full documentation. We plan to release the RTM within 2 weeks or so.
Initially we had planned for this release to port our LLBLGen Pro runtime framework to Netstandard1.6 (.net core) however we postponed that port to Netstandard2.0 and will release that port in v5.3. We did spend considerable time on the port to Netstandard 1.6 hence v5.2 took longer than expected.

What's new in LLBLGen Pro v5.2

LLBLGen Pro Designer

Major new features

Model- and meta-data analysis. This analysis reports non-error issues and suggestions for the model and meta-data in the project in real time. Issues reported are e.g. duplicate FKs, an FK that's pointing to itself or e.g. if an FK field is shared among multiple FK constraints. Suggestions are e.g. how many meta-data elements don't have a mapping or that there are e.g. several PK-PK relationships in the model and they can be used to create inheritance hierarchies. All suggestions and warnings are reported with ready-to-use actions you can click on to directly execute an action (e.g. map all unmapped tables to entities). This system will be extended in the future with more analysis options.
Several Entity Framework Core enhancements: v1.1.x is now supported, Firebird is now a supported database, read-only entities are now supported.
.NETStandard 1.6 is now a supported platform, for Entity Framework Core and the DTO classes framework.
Creating a project now asks for 'Initial contents' for the project. The options are: 'Empty project' (which is equal to the Create Project in previous versions), 'Relational model data retrieved from a database' or 'Relational model data container for a database'. The last two starts their respective wizards after the project has been created so the user doesn't have to look for the option in the user interface.

Minor changes / fixes / new features

The project file loader now reliably reports where in the file an xml error or other error was found during project load, and if possible which XML snippet caused it. This can greatly help finding errors in the project file, e.g. after a bad merge in a source control system.
All drivers now construct the connectionstring with DbConnectionStringBuilder, which should provide more protection against badly formed connectionstrings due to userinput.
A typeconverter which was moved to a different namespace is now normally loaded and used in existing projects which refer to the typeconverter with the old namespace name. The typename of the typeconverter itself has to be the same for this to make it work. Example: Foo.MyTC is used as typeconverter in a project. This typeconverter is moved to the namespace Bar: Bar.MyTC. Loading the project will now automatically use Bar.MyTC and not throw an error anymore that a type converter (Foo.MyTC) is missing.
Various context menu additions for various nodes in the Project Explorer to make it easier to find how to reverse engineer project elements (e.g. Entities, Typed Views) from relational model data elements like Tables, Views. It will pick the first database in the project which has its sync source set to database, or if that's not available, set to 'mixed'. If none of the databases in the project have a sync source set to database or mixed, it will pick the first one. For more fine-grained control, the catalog explorer reverse engineer functionality is the best choice.
QuickModel now has a flyout panel with quick help regarding commands so users can quickly check what the syntax is of common commands without first having to read a lot of documentation. It's opened by a new button in the QuickModel user interface called Command Help
The settings Exclude orphaned element detected action and Non excludable orphaned element detected action have been moved from Entity Model / General to Entity Model / Model First Development, as they're only applied in model first actions.
For framework settings, the property grid now automatically displays the default source of a setting so it's easy to find back what default setting in the project settings is controlling a setting on an element.
For attribute definitions, a new macro has been added: {$FriendlyName} which converts the element's name into a friendly name, where PasCal cased name fragments are prefixed with a space, and underscores are removed and replaced by a space. Example: OrderNumber and Order_Number will both become Order number. You can use this in e.g. Display() attribute definitions on elements. The element name without a group name or parent container name is used.
It's again possible to add fields in bulk to a typed list inside the typedlist editor. This feature was present in v4.2 and earlier.
The derived element editor now allows keyboard and right-click behavior in the shape editor, to make it easier to remove and denormalize fields.
The number of errors/warnings/information messages are now displayed in the toolbar of the designer for a quick overview, including buttons to go quickly to the errors/warnings/information messages in the Errors & Warnings pane.

LLBLGen Pro Runtime Framework

New functionality / changes

Data Scope now has Async / Await variants of its methods so it can be used fully asynchronously.
A global flag has been added for SelfServicing to disable lazy loading with one setting: EntityBase.EnableLazyLoading. The flag is enabled by default (so lazy loading is enabled by default) for backwards compatibility.
Lots of string interpolation has been refactored to more efficient string usage, resulting in faster performance overall and even lower memory footprint/object pressure. LLBLGen Pro is now the fastest ORM, beating all microORMs too (using the plain sql API).
Size specification on Init() methods in the generated FieldInfoProvider and PersistenceInfoProvider classes have been removed, so merge conflicts will no longer occur when merging different code bases of generated code.
For new projects and projects which change the target framework to the LLBLGen Pro Runtime Framework, the default output type for TypedLists and TypedViews has been changed to PocoWithQuerySpecQuery. This was TypedDataTable. If you still want to use TypedDataTable as the outputtype for TypedLists and TypedViews, you can change the default in the LLBLGen Pro Runtime Framework settings in the Project Settings. The default choice is for QuerySpec and not Linq because typed views mapped onto stored procedure resultsets aren't supported in Linq but are supported with QuerySpec. For existing projects the old default is kept.
Linq/QuerySpec: Using 'Distinct' on a subquery which is used inside an 'IN()' clause is now used inside the subquery. This is due to an addition in the FieldCompareSetPredicate which now allows 'Distinct' in the subquery. For most databases the 'Distinct' in the subquery has no effect, but on e.g. Oracle it can help the optimizer to pick an index instead of doing a table scan.
AddCollectionForSave and AddCollectionForDelete are now part of IUnitOfWorkCore.
DataAccessAdapter.SaveTransaction(name) and Transaction.SaveTransaction(name) no longer throw an exception when name was already used within the same transaction, as all supported databases which do support savepoints support multiple savepoints with the same name in a single transaction so the check / limitation is unnecessary.
Entity field validation now doesn't report a precision overflow for decimals/floats/doubles with precision equal to scale and a value between (-1.0, 1.0) excluded. Even though the precision of a value in that interval, e.g. 0.55, is 3, the leading zero isn't taken into account by databases and therefore the runtime shouldn't either.

Entity Framework Support

Entity Framework Core
The following new features / changes have been made to the support for Entity Framework Core

Netstandard 1.6 support. When selected as target platform, the designer will generate a csproj file in the Visual Studio 2017 format, targeting Netstandard 1.6.
Setting to generate a field as a readonly property and a setting to generate readonly fields as readonly properties. Both use the Entity Framework Core 1.1.x feature of backing fields instead of auto-properties.
Firebird is now a supported database
Support for Read-only entities
Setting to select the CollectionType for a navigator which returns a set. It's now possible to specify as type IEnumerable<T> and ICollection<T> instead of the default List<T>.

How to download the beta

The beta is available under 'My Account' -> Downloads -> v5.2 on the website, after you've logged in, and works with any valid v5.x license. As it is a beta, some bugs/issues might be present.

We also released pre-release builds of the runtime libraries on Nuget for your convenience.

LLBLGen Pro v5.1 RTM has been released!

Tue, 22 Nov 2016 14:04:29 GMT

After 2 successful EAPs and a beta, we’re happy to release the RTM version of LLBLGen Pro v5.1! To see what’s new in detail please see the blogposts about EAP1, EAP2, and the Beta.

In short:

Entity Framework Core v1.x support. The LLBLGen Pro designer now supports Entity Framework Core v1.x, the new version of Microsoft's Entity Framework ORM. More information.
Plain SQL Api. The LLBLGen Pro Runtime Framework now lets you execute plain SQL statements using parameters directly onto the database, including fetching resultsets and projecting the resultsets to POCO classes. With this API you don't need a microORM on the side anymore for those situations where the runtime couldn't give you the right query. This API is also very fast: fetching a query is faster than most microORMs. More information.
Target hints. The LLBLGen Pro Runtime Framework now supports select target hints for SQL Server and MySQL in Linq and QuerySpec queries. More information.
Temporal (history) table support. The LLBLGen Pro Runtime Framework now supports temporal tables for SQL Server 2016 and higher/ DB2 , through Linq / QuerySpec (Select / fetch queries only). Temporal tables are an easy, friction free way to store previous versions of modified / deleted rows in a table. The temporal table support in LLBLGen Pro allows you to include older versions of entities when querying. More information.
Easier projections for QuerySpec and stored procedures. It's now easier to create projections of stored procedure resultsets onto POCO classes, all it takes is specifying the target type. More information.

The new runtime is of course lighting fast and beats most (micro)ORMs, including Dapper and Entity Framework Core. See the latest raw data-access benchmarks: See results from November 22nd using the latest RawDataAccessBenchmark code.

LLBLGen Pro v5.1 is a free upgrade for all v5 subscription owners.

LLBLGen Pro v5.1 Beta has been released!

Thu, 27 Oct 2016 09:23:00 GMT

LLBLGen Pro v5.1 beta is now available. It’s the last public build before RTM, which I hope will be within a week or two. Since EAP2 we’ve added the following features:

Entity Framework Core support in the designer
Newer versions of the UI controls in the designer for better HiDpi support
.NET 4.6.2 support for all supported ORM frameworks.

The headliner of course is EF Core support in the designer, which makes LLBLGen Pro the no.1. designer for Entity Framework Core projects. The EF Core support is for v1.0.1, and will be updated once Microsoft ships 1.1.0 RTM.

Entity Framework Core is supported on .NET Full and .NET Core, the generated code is generated with a csproj for .NET Full, but is easily converted to a .NET core project.

Almost there!

LLBLGen Pro v5.1 EAP 2 released!

Wed, 28 Sep 2016 12:50:13 GMT

Today we released the second ‘Early Access Program’ (EAP) build of LLBLGen Pro v5.1! Please see the previous post on the first EAP build and what’s included in that version. This version builds further on that foundation and includes new features (as well as all fixes made to EAP 1 and v5.0). Our EAP builds are ‘ready to release’ builds, with up to date documentation and are fully tested.

Updated packages (marked ‘Alpha-20160928’) have been pushed to Nuget as well for the folks who prefer referencing nuget packages.

What’s included in EAP 2?

EAP 2 has the following new features, which are again all part of the LLBLGen Pro runtime framework. I’ll link to the documentation for details on the topics discussed below.

Plain SQL API

Using an ORM in general doesn’t confront you with plain SQL queries, often the APIs in ORMs don’t even allow you to specify a plain SQL query. This is OK for most situations but there are occasions where you might need to run a hand-optimized plain SQL query or have to execute a SQL query which can’t be generated by the ORM query API.

Since the arrival of microORMs, which often have a plain SQL interface as their core query API, users of full ORM frameworks sometimes also use a microORM ‘on the side’ to perform these plain SQL queries when needed. While this might work, it also can be a bit of a problem as one can’t leverage aspects offered by the full ORM, like an active transaction or easy paging query generation.

As we were already working on an addition to our POCO projection pipeline, we thought: why not open the runtime some more to be able to project plain SQL resultsets to POCOs as well? (and of course be able to execute SQL statements which don’t return a resultset). This is the new Plain SQL API.

All methods have async/await variants of course.

Executing a SQL statement

Executing a SQL statement is meant to execute a non-resultset SQL query, e.g. to update or delete some rows. This is easy to do with the ExecuteSQL method.

   1: Guid newId = Guid.Empty;   2: using(var adapter = new DataAccessAdapter())   3: {   4:     var q = @"SELECT @p0=NEWID();INSERT INTO MyTable (ID, Name) VALUES (@p0, @p1);";   5:     var idVar = new ParameterValue(ParameterDirection.InputOutput, dbType: DbType.Guid);   6:     var result = adapter.ExecuteSQL(q, idVar, "NameValue");   7:     newId = (Guid)idVar.Value;   8: }

The example above inserts a row into MyTable and returns the generated NEWID() Guid through an output parameter. It passes the value ‘NameValue’ as parameter value to the query as well. There’s a flexible way to specify parameters with just the values as arguments of ExecuteSQL and if needed, you can define a ParameterValue instance with parameter specifics, like direction, DbType, length, precision, scale etc. This is also the mechanism used to obtain the output parameter value after the query has been completed.

Fetching a resultset

Of course you can also fetch a resultset and project to POCO classes using the Plain SQL API. This is done using the FetchQuery method. See the following example:

   1: List<Customer> result = null;   2: using(var adapter = new DataAccessAdapter())   3: {   4:     result = adapter.FetchQuery<Customer>(   5:                     "SELECT * FROM Customers WHERE Country IN (@p0, @p1) ORDER BY CustomerID DESC",   6:                     "USA", "Germany");   7: }

In the example above a query on the Customers table with a WHERE clause using two parameter values is projected onto the POCO class Customer. This is all nice and works great, but as the API is part of a full ORM, there’s more: we can leverage systems in the full framework. Here’s the same example again, but this time it utilizes the Resultset caching system in LLBLGen Pro and it also offers paging query creation for a specific offset / limit combination:

   1: List<Customer> result = null;   2: using(var adapter = new DataAccessAdapter())   3: {   4:     // fetch first 5 rows, cache resultset for 10 seconds.   5:     result = adapter.FetchQuery<Customer>(new PlainSQLFetchAspects(limit: 5,    6:                     cacheResultset: true, cacheDuration: new TimeSpan(0, 0, 10)),    7:                     "SELECT * FROM Customers WHERE Country IN (@p0, @p1) ORDER BY CustomerID DESC",   8:                     "USA", "Germany");   9: }

Here the query uses an instance of PlainSQLFetchAspects which allows you to specify an optional limit, offset and caching specifics. In the example above a limit of 5 is specified, no offset (so the first 5 rows of the resultset). Additionally it specifies that the resultset has to be cached for 10 seconds. This means that if an exactly the same query is executed within 10 seconds, it will pull the resultset from the cache instead of obtaining it from the database again. It checks the compatibility mode of the adapter used and will create a paging query matching it, e.g. an OFFSET query on SQL Server 2012+ and a CTE on SQL Server 2005-2008.

The projections to POCO classes are done using property name-column name comparisons: every property in the specified POCO (here: Customer) with a name which is equal (case insensitive comparison) to a column in the resultset will receive the value in that column.

Executing a stored procedure to fetch a resultset is similar: instead of a SELECT… query you simply specify the EXEC proc_name parameters statement as the SQL statement. See the example below:

   1: List<Order> result = null;   2: using(var adapter = new DataAccessAdapter())   3: {   4:     var result = adapter.FetchQuery<Order>("EXEC CustOrdersOrders @p0", "CHOPS");   5: }

This example fetches the resultset returned by the stored procedure CustOrdersOrders while passing in a parameter, namely the CustomerID.

Transactions

As the plain SQL queries are executed through the same pipeline as normal entity queries are, a plain SQL query can participate in a transaction which also contains entity activity and you can e.g. use the known transaction mechanism to run multiple plain SQL statements in one transaction.

Is it fast?

You bet! Using my own RawDataAccessBencher system, the Plain SQL API is one of the fastest APIs to fetch data on .NET, beating many micro ORMs which are considered ‘much faster than full ORMs’: Results from 21st September 2016 (‘Poco with Raw SQL’ results). This makes LLBLGen Pro Runtime Framework one of the fastest ORMs on .NET, only Linq to DB is faster. The regular pipeline was already one of the fastest, with the Plain SQL api we made the final small step to reach the absolute holy grail: be (almost) as fast as handwritten materializer code.

One key aspect of why the Plain SQL API is a bit faster than the regular pipeline is that the Plain SQL api can cut some corners in the projection: it doesn’t have to run type converters if needed, as type converters aren’t supported in the Plain SQL API. This frees the pipeline from first boxing the row from the datareader in an object array. Type converters are a feature of the LLBLGen Pro runtime framework where you can transparently convert one type to another and back when persisting data to the database or reading data from the database (e.g. convert a byte[] to a bitmap class, convert a string to a number, anything is possible).

What about SQL Injection?

Plain SQL APIs have a downside: a developer can think s/he’s ‘smart’ and think “I’ll just embed this value in the query string, no need for a parameter!”, and with that opening the application to external SQL Injection attacks. With a Plain SQL API it’s not possible to check for this. What we have done instead are two things: add a specific tracer to log all plain SQL queries in a separate output, and add a way to disable the API entirely. (More info about these two options in the documention)

This gives options to our users: if they don’t want their developers to use the Plain SQL API, they can simply add a small code snippet and it will disable the API. If they instead want to monitor whether their developers cut corners by embedding values in the plain SQL statements (or use the API at all!) they can simply add a tracer through .NET’s config system and get the output in a file or other storage mechanism and check the output.

Easier projections to POCO classes in QuerySpec

We added an easier way to project QuerySpec queries to POCO classes. Besides the regular way to project a resultset to a POCO using ToValue<T>() calls for every property, it’s now also possible to simply specify the field list and the resulting type. Using the same property name-column name system as with the Plain SQL API, it’s no longer necessary to specify each field projection individually. See the example below:

   1: var qf = new QueryFactory();   2: var q = qf.Create()   3:             .Select<CustomerDTO>(CustomerFields.Country,   4:                 Functions.IIF(CustomerFields.Country == "USA", CustomerFields.ContactName,    5:                         CustomerFields.ContactTitle).As("CheckValue"),    6:                 CustomerFields.ContactName,   7:                 CustomerFields.ContactTitle);

The example above fetches 3 fields from the target Customer is mapped on and utilizes a function call (IIF, which results in a CASE statement) as well. The resultset is then projected to instances of CustomerDTO.

Easier projections to POCO classes of stored procedure resultsets

Projecting a resultset of a stored procedure call to a POCO class was already doable without much effort in the designer, but if you wanted to project a resultset to a POCO type you have written by hand it was rather verbose. We solved this with the same projection system as above, which results in simple code:

   1: List<CustomerDTO> customers;   2: List<OrderDTO> orders;   3: using(var adapter = new DataAccessAdapter())   4: {   5:     using(var query = RetrievalProcedures.GetCustomersAndOrdersOnCountryCallAsQuery("Germany"))   6:     {   7:         using(var reader = adapter.FetchDataReader(query, CommandBehavior.CloseConnection))   8:         {   9:             customers = adapter.FetchProjection<CustomerDTO>(reader);  10:             reader.NextResult();  11:             orders = adapter.FetchProjection<OrderDTO>(reader);  12:             reader.Close();  13:         }  14:     }  15: }

The example above calls a stored procedure, which is mapped as a call in the designer, and fetches two resultsets from this procedure, as it returns two resultsets. It does this by using the FetchDataReader API and projects the resultsets using the new FetchProjection<T> method.

That’s it for now, more to come in the near future as we’re not done with v5.1 yet, stay tuned!

LLBLGen Pro v5.1 EAP1 released!

Tue, 30 Aug 2016 13:51:24 GMT

Today we released our first ‘Early Access Program’ build for LLBLGen Pro v5.1! When we moved to subscriptions (with perpetual licenses) when we released v5.0, the one thing I wanted to get rid of was the long delays between versions: no more 1.5-2 years of development to a massive release, but smaller releases which are given to the users quickly. So here is the first release of that. Additionally, we did a lot of work to make it release-ready. This means that the EAP build is a release like any other final release, including up-to-date documentation and fully tested. This is another big step for us, so we can switch an EAP build to ‘RTM’ at any time.

In the coming months we’ll release more builds till we reach RTM.

What’s included?

This first EAP release contains the following features. We’re focusing on our state of the art ORM framework this time around for most of the features planned for v5.1 RTM.

Temporal (history) table support (SQL Server 2016 / DB2 10)

For select / fetch queries, the LLBLGen Pro runtime framework now supports temporal tables. A temporal table is a table with a coupled history table which is managed by the RDBMS: an update or delete of a row will copy the original row to the history table with two date markers to signal the period in which this row was valid. For more information about temporal tables in e.g. SQL Server, see this article.

Temporal tables offer a transparent way to work with history data. This means you can now use temporal table predicates directly in Linq and QuerySpec (our fluent query API) queries to query on the current data but also on history data. See this example:

   1: var q = from e in metaData.Employee   2:                             .ForSystemTime("BETWEEN {0} AND {1}",    3:                                             fromDate, toDate)   4:         where e.EmployeeId == 1   5:         select e;

Here a Linq query is defined which will query for the employee data of the employee with id ‘1’, and all rows valid between fromDate and toDate are included. This means that if the employee data of this particular employee was updated between these two dates, the original data which was updated will be included in the resultset as well.

On IBM DB2, LLBLGen Pro also supports Business Time temporal table predicates, something which isn’t supported by SQL Server 2016.

Table / View hints (SQL Server) and Index hints (MySQL)

To specify a hint for the RDBMS query optimizer has been a requested feature for a long time, but I never found a proper way to make it easy to specify. With the temporal table support, the same mechanism can be used for specifying hints for table / views, in the case of SQL Server, and indexes, in the case of MySQL. All other databases which support hints (Oracle and DB2 come to mind) aren’t supported here, as they force the hints to be present as comments in the projection of the SELECT statement, and the hint system works with hints specified on elements in the FROM clause. This isn’t that bad however, as hints on Oracle and DB2 are heavily discouraged by the vendors of these databases so it’s unlikely we’ll add support for these particular hints later.

To specify a table / view hint in a Linq query (or QuerySpec query), you simply call an extension method with the hint as argument as shown in the following example (SQL Server):

   1: var q = from c in metaData.Customer   2:                                 .WithHint("NOLOCK")   3:                                 .WithHint("FORCESEEK")   4:         join o in metaData.Order on c.CustomerId equals o.CustomerId    5:         where o.EmployeeId > 4   6:         select c;

Here the target mapped by the entity ‘Customer’ will receive two hints specified in the SQL Query, namely ‘NOLOCK’ and ‘FORCESEEK’. The target mapped by the entity ‘Order’ doesn’t receive these hints.

Hints are a last resort to optimize a query and in general they should be left alone, but in cases where it can help a great deal, it’s good that an ORM Framework supports facilities to specify hints so the developer doesn’t have to fall back to raw in-lined SQL or a stored procedure.

More to come!

The .NET support black hole

Thu, 23 Jun 2016 12:49:56 GMT

Today I ran into a bit of an issue. A work-item for LLBLGen Pro v5.1 is to support all the new features of SQL Server 2016. One of the features of SQL Server 2016 is ‘Always Encrypted’. You can enable this feature through the connection string, and after that all data-access is encrypted, no further coding needed. As this is a connection string setting, it’s enabled in every ORM out there out of the box, also in ours. That’s of course not the problem. The problem is adding more control over this feature to the developer writing code which targets SQL Server 2016.

Starting in .NET 4.6, the SqlClient API offers a way to specify when and when not to encrypt using SQL Server 2016. This is done through a setting in SqlCommand’s constructor: you can specify a SqlCommandColumnEncryptionSetting value which gives you control over when to encrypt and when not to encrypt, which could greatly enhance performance if you just partly encrypt your catalog.

There’s something odd though: Although SqlCommand has a property for this, ColumnEncryptionSetting, it’s read-only. Its backing variable is set through the constructor. Now, why is this a problem, you ask? Well, unless your code is creating SqlCommand instances directly, you can’t set the setting for a given SqlCommand instance: if you use DbProviderFactory, and most ORMs do, or if you use CreateCommand() on the connection object, you can’t set the setting. You can only set it if you directly use the constructor.

Any database-generic code out there uses either DbProviderFactory or the CreateCommand() method on the connection object, and thus can’t use this feature.

The problem

Looking at this, somewhat terribly designed SqlCommand API, I wondered: “Ok, there’s a serious problem with this API, where can I give feedback so they can fix it?”. I couldn’t answer that. With a product you purchase from a vendor, you can do to that vendor’s support channel, ask them what they think should be done to get this resolved and you eventually reach a conclusion, but here, I have literally no idea.

With .NET Core, most development regarding .NET within Microsoft is focused on that, and a lot of airtime is given to that on GitHub, blogposts etc. But .NET full, e.g. v4.6.2, where do you go with an issue like this? Connect? Mail someone within Microsoft, hoping they’ll route it to some person who won’t delete it right away and look at it? About Connect I’ll be short: no way in hell am I going to spent another second of my time on this planet in that crappy system. Not only does the .NET team not reply to any issues there, I still have some open issues there which are years old and no-one bothers to answer them. It’s like writing the problem into some text file and never look at it again, same result.

About emailing someone within Microsoft: that might work, but it also might not. I happen to know some people within Microsoft and I’m sure they’ll at least read the email, but it’s a silly way to give feedback: here we have a mega-corporation which makes billions of dollars each month, says to be a developer focused company and you have to email your question to some employee and hope to get answers? How fucked up is that!

Now, my technical issue with SqlCommand is not something everyone will run into. That’s also not the point. The point is: if there’s an issue with the .NET BCL/API, there should be a clear path to the teams working on this codebase to give them feedback, report issues and get things fixed. Today there’s none. Oh of course, if things related to .NET Core pop up, we can open an issue on GitHub and start the conversation there, but this isn’t related to .NET Core: SqlClient on .NET core doesn’t contain any encryption related code (as far as I can tell, .NET Core’s SqlCommand lacks code related to encryption which is in the SqlCommand class in the reference source), so this issue is out of scope for .NET Core. (As an aside, the SqlClient team isn’t really responsive on .NET Core issues either, e.g. #1039 or #3480)

Microsoft seriously has to get things fixed in this area. We .NET developers now have no way to give feedback to them regarding APIs we have to work with every day. It’s not sustainable nor reliable. I don’t need the email address of the SqlClient team, I need a publically available system where I can file issues and Microsoft can respond swiftly to them so I can tell my clients and customers why things are the way they are. Which is how we do things in the ISV world: give support to our paying customers quickly and fully so they can help their customers quickly and fully. You know, as professionals.

What Microsoft currently ‘offers’ is a sign which says ‘back in 5!’ although the sign is up there for a long time.

LLBLGen Pro v5.0 RTM has been released!

Tue, 26 Apr 2016 13:32:38 GMT

After 1235 commits and 20 months of full time development, the work is done and we’ve released LLBLGen Pro v5.0! To see what’s new, please go to the official ‘New features’ page on our website, or read about them on this blog in the Beta announcement post.

I’m very happy with this release and how it turned out. I’m confident you’ll be too. We have much more planned in the upcoming releases, so stay tuned!

To all the beta-testers, support team and others who have helped us to make it the best release we’ve ever done: Thank you!

Frans Bouma's blog

LLBLGen Pro v5.6 has been released!

LLBLGen Pro v5.5 feature highlight: query batching

LLBLGen Pro v5.5 feature highlight: Window functions

LLBLGen Pro v5.5 has been released!

LLBLGen Pro v5.5 beta has been released!

What’s new in the LLBLGen Pro Runtime Framework?

New functionality / changes

Small changes

What’s new in the Designer?

Major changes / fixes / new features

Minor changes / fixes / new features

LLBLGen Pro v5.4 has been released!

Designer

Major changes / fixes / new features

Minor changes / fixes / new features

LLBLGen Pro Runtime Framework

Let’s add a photo mode to Wolfenstein II: The New Colossus (PC)

Table of Contents

Introduction

Getting started, the tools we need

What information do we need?

The camera

Game pause

Hide HUD and other on-screen cruft

To recap…

Setting things up

Memory allocation in windows processes

Finding the FoV

Finding the camera

Hide HUD using the g_showhud variable

Game pause / Timestop

Reverse engineering and what it is all about

Doom to the rescue

“Dude, this is just plain luck”

What else is there to find?

Creating the camera tools

Configuring IGCS to hook into the .exe

(Raw)Input hooks

Game exe hooks

Setting up IGCS to write its camera data

IGCS’ mainloop

Testing the camera tools

Determining hard-coded addresses directly from the exe’s bytes

Compiling and testing the release build

Conclusion

Optimizing memory usage

Why optimize memory usage and when?

Profile, Analyze, Fix

When to optimize?

Visual Studio Memory Profiler

Optimizing the Linq provider

Avoid params arrays

Avoid StringBuilder.AppendFormat()

Pre-size data-structures if you add data immediately afterwards

Initialize members lazily

Avoid Linq-to-Objects if necessary

Optimizing the SQL generator

End results

.NET (Micro)ORM fetch benchmark results and the fine details

The graphs

Set fetches, non-change tracking

Set fetches, change tracking

Individual Fetches, non-change tracking

Individual fetches, change tracking

Eager loading, 3-node split graph

Eager loading, 3-node split graph, async

Conclusion

LLBLGen Pro v5.3 released!

LLBLGen Pro v5.3 Beta has been released!

What's new in the LLBLGen Pro designer

What's new in the LLBLGen Pro Runtime Framework

What's new in the Entity Framework support

What's new in the NHibernate support

LLBLGen Pro v5.3 EAP Released!

LLBLGen Pro v5.2 has been released! (and first quickstart video posted)

LLBLGen Pro v5.2 Beta has been released!

What's new in LLBLGen Pro v5.2

LLBLGen Pro Designer

LLBLGen Pro Runtime Framework