Manuel Abadia's ASP.NET stuff

Binding a DropDownList to multiple properties and to nested properties

Manu — Mon, 22 Dec 2008 23:35:20 GMT

All controls that inherit from ListControl (BulletedList, CheckBoxList, DropDownList, ListBox and RadioButtonList) have some annoying behavior in common:

You can only bind the control to a single property using the DataTextField property. For example, if you have a Customer class with properties Id, Name, Surname, etc and you want to show the full name of a customer in any of the ListControls, you have several options:

When you get the data from the database, return another column with the name and surname.
Create a property (f.e. FullName) that returns what you want to show.
Use a DataSet and add a computed column.
Manually add ListItems with the appropriate values performing the data bind manually.
Create a simple class that will be bound (instead of the original one) just to fix the problem.

The truth is that I don’t like any of those options, all of them are hacks.

For nested properties you have the similar problems. You can bind to Id or Name, but you can’t bind to Address.City or Address.Zip.

Today I faced this problem again and I decided to investigate a more innovative solution. The problem itself is in the controls that inherit from ListControl that are very strict in the binding options. So I took a look of how the controls perform the binding to see if I could do anything to overcome those limitations. At first sight I thought I was loosing my time but after a deeper study I found that it was amazingly easy to fix those problems in a very sleek way.

The data binding of the controls is performed in the PerformDataBinding method. The code at ListControl.PerformDataBinding uses DataBinder.GetPropertyValue to retrieve the value to show in the control if you set the DataTextField property. However, DataBinder.GetPropertyValue doesn’t handle nested properties. DataBinder.Eval is a better choice because it does handle nested properties. So using it we solve one problem. The other problem can be solved easily as well. As we have two properties that control how the Text of each ListItem will be extracted and shown from the data source, with a bit more of effort we can handle the new functionality. I have come up with a very intuitive solution:

DataTextField allows one or more properties (that can be nested if needed) separated by commas.
DataTextFormatString can be used to control the formatting of the items, using {0}, {1}, {2}, {3}… as the value of the properties specified with DataTextField.

The code that performs the data binding is:

///
<summary>Binds the data to the control.</summary>

/// <param name="ctrl">control
to bind.</param>

/// <param name="dataSource">data
to bind to the control.</param>

public static void PerformDataBinding(ListControl
ctrl, IEnumerable dataSource)

{

    // resets the selected
item

    ctrl.SelectedIndex = -1;

    ctrl.SelectedValue = null;

    if (dataSource != null) {

        bool fieldInfoSet = false;

        bool dataTextFormatStringSet = false;

        // checks
if we have to clear the existing items

        if (!ctrl.AppendDataBoundItems) {

            ctrl.Items.Clear();

        }

        // if
the data source is a collection, sets the capacity

        ICollection soureceCollection = dataSource as ICollection;

        if (soureceCollection != null) {

            ctrl.Items.Capacity = soureceCollection.Count + ctrl.Items.Count;

        }

        // save
if the data text field of data value field has been set

        if ((ctrl.DataTextField.Length != 0) || (ctrl.DataValueField.Length != 0)) {

            fieldInfoSet = true;

        }

        // save
if the data text format string has been set

        if (ctrl.DataTextFormatString.Length != 0) {

            dataTextFormatStringSet = true;

        }

        // iterates
through the data source creating the ListItems

        foreach (object obj in dataSource) {

            ListItem item = new ListItem();

            if (fieldInfoSet) {

                if (ctrl.DataTextField.Length > 0) {

                    if (ctrl.DataTextField.IndexOf(',') == -1) {

item.Text = DataBinder.Eval(obj,
ctrl.DataTextField, ctrl.DataTextFormatString);

                    } else {

                        //
if the DataTextField property has a list of fields, get them to create the text of
the item

                        string[] fields = ctrl.DataTextField.Split(new string[] { "," },
StringSplitOptions.RemoveEmptyEntries);

                        object[] values = new object[fields.Length];

                        for (int i = 0; i < fields.Length; i++) {

    string field = fields[i];

    values[i] = DataBinder.Eval(obj,
field);

                        }

item.Text = String.Format(ctrl.DataTextFormatString,
values);

                    }

                }

                if (ctrl.DataValueField.Length > 0) {

                    item.Value = DataBinder.Eval(obj,
ctrl.DataValueField, null);

                }

            } else {

                if (dataTextFormatStringSet) {

                    item.Text = string.Format(CultureInfo.CurrentCulture,
ctrl.DataTextFormatString, new object[] { obj });

                } else {

                    item.Text = obj.ToString();

                }

                item.Value = obj.ToString();

            }

            ctrl.Items.Add(item);

        }

    }

}

As we need to call this data binding code instead of the one from ListControl I needed to subclass all controls that inherit from ListControl in order to fix them. So now I have BulletedListEx, CheckBoxListEx, DropDownListEx, ListBoxEx and RadioButtonListEx.

I have created a simple example showing how to use them. Imagine that an Item can be supplied by a set of suppliers, and each supplier sells the item with a different price. The following image shows the properties of the 3 entities of the sample:

I have created a web page with a DropDownList with some items, and when you select an item in the DropDownList you will see a ListBox with the all suppliers of that item with the associated price for the item. Notice how I show the Id and the Name in the DropDownList and how I show the price, supplier Id and supplier name in the ListBox:

The ASPX of the page follows:

<body>

    <form id="form1" runat="server">

    <div>

    </div>

        Choose an item to see the suppliers:<br />

        Items:<br />

        <manu:DropDownListEx ID="DropDownListEx1" runat="server" AutoPostBack="True" Width="300px"

            DataTextField="Id,
Name" 

            DataTextFormatString="[{0}]
- {1}" 

            DataValueField="Id" 

            OnSelectedIndexChanged="DropDownListEx1_SelectedIndexChanged" AppendDataBoundItems="True">

            <asp:ListItem Selected="True" Value="0">Select
an Item</asp:ListItem>

        </manu:DropDownListEx><br />

        Price and suppliers:<br />

        <manu:ListBoxEx ID="ListBoxEx1" runat="server" Width="300px"

            DataTextField="Price,
Supplier.Id, Supplier.Name" 

            DataTextFormatString="{0}
- [{1}] {2}" 

            DataValueField="Supplier.Id">

        </manu:ListBoxEx>

    </form>

</body>

As you can see, the solution is really clean and elegant, and backward compatible with the current controls that inherit from ListControl.

That’s all for now,
Merry christmas!

ListControlsEx_bin.zip (6.53 KB)

ListControlsEx_sample.zip (33.75 KB)

Search Engine Optimization - Moving View State to the bottom of the page

Manu — Thu, 04 Dec 2008 19:26:16 GMT

One of the greatest problems when trying to optimize an ASP.NET page to be more search engine friendly is the view state hidden field. Most search engines give more score to the content of the firsts thousands of bytes of the document so if your first 2 KB are view state junk your pages are penalized. So the goal here is to move the view state data as down as possible.

I have seen some approaches to solve this problem rewriting the final HTML code of the response. While this approach works I think that it wastes some precious processor cycles that can be used to do other things. So I needed a way to do the same thing without wasting that CPU time. After some large reflector sessions I found a way to do it. My method uses the ASP.NET Control Adapter Architecture.

A control adapter is a class that can be used to control the HTML generated by the control it adapts. Since the Page class is the responsible of rendering the view state hidden field (Page.BeginFormRender calls Page.RenderViewStateFields), an adapter for the Page is needed. However, the view state hidden field plays a key role in the ASP.NET infrastructure (for example, the Page.IsPostBack property checks if the view state hidden field has been posted) and it is difficult to modify the associated HTML.

A PageAdapter has a method called GetStatePersister() that returns an object that inherits from PageStatePersister. The PageStatePersister is called when it is time to load and save the view state. There are 2 classes that inherit from PageStatePersister: HiddenFieldPageStatePersister and SessionPageStatePersister. The first one is the default, which stores the view state in the hidden field called __VIEWSTATE. The second one stores the view state in the session. So, we can easily create a custom PageStatePersister to control the view state load and save process. The big problem is how to create the hidden view state field before the closing form tag while being a fully transparent solution. After some tries I came up with a solution that I was happy with.

I realized that it was impossible to completely remove the view state hidden field from the top of the page, because it plays a key role in the ASP.NET infrastructure. However, with any custom page state persister the ASP.NET infrastructure renders at least an empty view state hidden field of only 70 bytes:

<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="" />

My page adapter adds a hidden field to the bottom of the form called __SEOVIEWSTATE with the actual view state data, and the only limitation that it has it is that you can not use <% %>expressions directly inside the asp.net form. However, this restriction can be easily avoided putting the <% %>expression in a PlaceHolder control or inside another control. For an in-depth explanation of this limitation take a look here.

Let’s see an example of the adapter in action. The following ASP.NET page:

<%@
Page Language="C#" AutoEventWireup="true" CodeBehind="Default.aspx.cs" Inherits="SEOViewStateAdapterTest._Default" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >

<head runat="server">

    <title></title>

</head>

<body>

    <form id="form1" runat="server">

    <div>

        Enter a message: <br />

        <asp:TextBox ID="txtMessage" runat="server"></asp:TextBox><br />

        <asp:Button ID="bSaveMessage" runat="server" Text="Save
Message" onclick="bSaveMessage_Click" /><br />

        <asp:Label ID="lMessage" runat="server"></asp:Label><br />

        <p>Page
generated at <asp:PlaceHolder ID="PlaceHolder1" runat="server"><%=
DateTime.Now.ToString("hh:mm
dd/MM/yyy") %></asp:PlaceHolder></p>

    </div>

    </form>

</body>

</html>

With the associated code:

using System;

using System.Collections.Generic;

using System.Web;

using System.Web.UI;

using System.Web.UI.WebControls;

namespace SEOViewStateAdapterTest

{

    public partial class _Default : System.Web.UI.Page

    {

        protected void Page_Load(object sender,
EventArgs e)

        {

            if (!Page.IsPostBack) {

                ViewState["previousMessage"] = txtMessage.Text;

            }

        }

        protected void bSaveMessage_Click(object sender,
EventArgs e)

        {

            lMessage.Text = String.Format("The
current message is '{0}'. The previous message was '{1}'", txtMessage.Text, (string)ViewState["previousMessage"]);

            ViewState["previousMessage"] = txtMessage.Text;

        }

    }

}

after a couple of postbacks, without using the adapter, the HTML looks like this:

<!DOCTYPE
html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >

<head><title>

</title></head>

<body>

    <form name="form1" method="post" action="Default.aspx" id="form1">

<div>

<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUJc01l[...]VwP+cfdSWI6Q==" />

</div>

<div>

    <input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="/wEWAwKb4uenCgK/1s7/DwKf8MMPfiUvZtKPSXk//XdxkLooz8QDI0Y=" />

</div>

    <div>

        Enter a message: <br />

        <input name="txtMessage" type="text" value="Message
2" id="txtMessage" /><br />

        <input type="submit" name="bSaveMessage" value="Save
Message" id="bSaveMessage" /><br />

        <span id="lMessage">The
current message is 'Message 2'. The previous message was 'Message 1'</span><br />

        <p>Page
generated at 08:13 04/12/2008</p>

    </div>

    </form>

</body>

</html>

and after a couple of postbacks, using the adapter, the HTML looks like this:

<!DOCTYPE
html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >

<head><title>

</title></head>

<body>

    <form name="form1" method="post" action="Default.aspx" id="form1">

<div>

<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="" />

</div>

<div>

    <input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="/wEWAwL+raDpAgK/1s7/DwKf8MMPyF7nqN1AbwNwFBq8OAjEAQorsyo=" />

</div>

    <div>

        Enter a message: <br />

        <input name="txtMessage" type="text" value="Message
2" id="txtMessage" /><br />

        <input type="submit" name="bSaveMessage" value="Save
Message" id="bSaveMessage" /><br />

        <span id="lMessage">The
current message is 'Message 2'. The previous message was 'Message 1'</span><br />

        <p>Page
generated at 08:11 04/12/2008</p>

    </div>

    <input type="hidden" name="__SEOVIEWSTATE" id="__SEOVIEWSTATE" value="/wEPc01lc3[...]CdNY6AtgigHvU=" /></form>

</body>

</html>

In order to use the adapter, you have to add a reference to the assembly and add a file called SEOViewStateAdapter.browser (the name of the file does not matter. The extension needs to be the same. Or also you could merge the contents to another file if you already have one) to the App_Browsers folder. The content of the file should be:

<browsers>

    <!-- use the adapters
for all browsers -->

    <browser refID="Default">

        <controlAdapters>

            <!--
ths adapter is used to save the view and control state at the bottom of the form so
the page is more friendly to search engines -->

            <adapter controlType="System.Web.UI.Page" adapterType="Manu.Web.Adapters.SEOFriendlyViewStatePageAdapter,
SEOViewStateAdapter" />

        </controlAdapters>

    </browser>

</browsers>

That’s all. Enjoy!

SEOViewState.zip (29.67 KB)

XHTML Validation Module Enhancement

Manu — Wed, 13 Aug 2008 23:12:04 GMT

I have been using Josh Twist’s XHTML validation module for a while and I have added a couple of small enhancements I have found useful:

It works without internet access as I included the required DTDs as embedded resources. I usually have internet access when I’m using it, but it is a big problem if I don’t.
It doesn’t validate ASP.NET trace output if it is present in the page. This is checked using the trace configuration section. I have not found a way to know if trace output is enabled in the current request.
I have added an attribute called showErrorsOnly to the validatorModule element to avoid calling the Renderer.Render method if there isn’t any error. That way, if the page validates successfully no message is shown.

Download validator_binaries (42,78 KB)

Downlaod Validator_source code (44,04 KB)

Update to my NHibernate Custom Membership and Role Providers

Manu — Fri, 01 Aug 2008 10:10:15 GMT

The changes from v1.2 to the current one (v2.0) are:

Now using NHibernate 2.0.0 CR1.
Fixed a bug that caused the provider to look for the configuration settings only in the nhibernate config section. If you configured NHibernate using the hibernate-configuration section, it didn't work.
Fixed a bug that prevented hashed passwords to work without supplying a password answer.
Some code refactoring/clean up.
Removed all FXCop warnings.
Changed the mapping generation for the NHCustomMembershipProvider. In NHiberante 2.0.0 CR1, it is not possible to create mappings programatically as before, so now a xml document with the mapping is created.
Fixed a bug in ChangePassword and ResetPassword methods that made the stored password answer useless if the provider was using hashed format with password salt.

From some questions/emails I've got, it seems that some people aren't using the provider the way it was designed. There is no need to modify the code in the provider to support custom fields for your Users or Roles. I have created a sample that uses all the fields in the membership provider and adds some custom fields to show how the provider should be used. The sample also uses the CreateUserExWizard control in order to create an user.

As always, feedback is welcome. It has been tested only in SQL Server 2005. Let me know if you try it in other databases.

The binaries and source code can be downloaded here

World Rally emulation completed

Manu — Mon, 12 May 2008 09:13:01 GMT

This post is first in English and then in Spanish
Esta noticia está primero en inglés y luego en español

World Rally emulation is complete. I implemented priorities (that were trickier than I thought), shadows/highlights and a few missing bits in the video hardware. Here are some screenshots of the finished driver:

The game is fully playable from beginning to end without any problem as far as I can tell. I thought there was a bug in the video hardware emulation somewhere when I saw this:

So I plugged my PCB and compared it to the driver:

The problem is also in the original arcade game. Because of the way an arcade monitor works, the problem is nearly unnoticeable in the original game. I also checked if the original PCB had the same “shadow effect” for the tiles:

As you can see, the hardware works that way:

Something curious about the protection... Javier told ElSemi that the protection of this game took 8 months of work, so imagine how complicated it was… even the dallas has some code that performs some pseudorandom dummy accesses to the shared RAM to make black box attacks even more difficult.

To clarify a question about the other protected games, having the World Rally dallas code does not help to emulate the protection of them. As MAME now has a DS5002FP core and the other games are almost fully emulated, if we get the dallas code for a game, it will be playable quickly.

Finally, I want to thank to all the people that made this possible. It was cool to be part of this. It has brought me some good memories and healed my wounds with Gaelco.

Update: Gaelco made public the ROMs for World Rally. You can get them from their web page

For reference here are all the posts about World Rally emulation:

Part 1
Part 2
Part 3
Part 4

La emulación del World Rally está completada. He implementado las prioridades (que han sido más difíciles de lo que pensaba), los focos y sombras, y algún detalle que faltaba del hardware gráfico. A continuación se muestran unas pantallas del driver:

Es completamente jugable sin que haya podido observar ningún problema. Pensé que había algún fallo en la emulación del hardware gráfico cuando vi esto:

Así que enchufé mi placa y la comparé al driver:

Como se observa, el problema también está presente en el juego original. Debido a la forma en la que funciona un monitor de recreativa, el pequeño fallo gráfico es prácticamente inapreciable en el juego original. También comprobé si el juego original hacía el mismo “efecto de sombra” de los puentes:

Como se puede apreciar, el hardware funciona de esa manera:

Algo curioso sobre la protección… Javier le dijo a ElSemi que la desarrollar protección del juego llevó 8 meses de trabajo. Inmaginad cómo de complicada es la protección… Incluso el dallas tiene código para realizar accesos pseudoaleatorios a la memoria compartido con el único motivo de hacer que los ataques de tipo caja negra sean incluso más difíciles.

Sobre la pregunta de si tener el código del dallas para el World Rally sirve para emular la protección de los otros juegos, la respuesta es que no. Se necesita el código de cada uno para hacerlo funcionar. Como ahora el MAME ya dispone de un emulador de DS5002FP y los juegos de Gaelco protegidos están ya emulados completamente, una vez que obtengamos el código del dallas para un juego, será jugable rápidamente.

Por último, me gustaría darle las gracias a toda la gente que ha hecho esto posible. Ha sido muy interesante poder formar parte de esto. Me ha traído muy gratos recuerdos y ha cerrado viejas heridas con Gaelco.

Actualización: Gaelco ha hecho públicas las ROMs del World Rally. Puedes obtenerlas desde su página web

A modo de referencia, aquí están todas las noticias sobre la emulación del World Rally:

Parte 1
Parte 2
Parte 3
Parte 4

Playable status reached

Manu — Sat, 10 May 2008 18:52:23 GMT

This post is first in english and then in spanish.
Esta noticia está primero en inglés y luego en castellano.

Javier sent us (via ElSemi) detailed information about how the encryption process worked. However, he told us that the sheets of paper that contained the encryption info were a bit difficult to read in some places (remember that this protection was designed 15 years ago) so probably there was some mistakes in the excel file he sent us. ElSemi and I tried to add it with the information he gave us but we didn’t have success. Nicola Salmoria did an excellent work (as always) consolidating the information and obtaining the missing information. He generated a working decryption function that not only works for World Rally, but also for Squash and Thunder Hoop.

Mike Coates also completed the interface to get the data from a World Rally PCB, so he was able to supply the encrypted/decrypted data that was needed to get the specific details to decrypt World Rally properly.

Special mention to Andreas Naive too. The high level information Javier told us a few days ago was already discovered by Andreas and posted in his page a couple of months ago. Before Javier sent us detailed information about how the encryption worked I contacted Andreas Naive about the encryption and he told me that he didn’t have free time at the moment to look at it. However he told me an intuition he had about the algorithm:

"I remember that my last feeling was that to decipher each 16 bits block, it was done in 3 chunks: a first 6 bits chunk and then two 5 bits chunks, each of them based in a/some bits of the first chunk. I don't remember exactly which ones, but the first chunk was the one with a simple structure (based on the tables I published), while the other two were the ones that shown a more complex structure (and carry effects)".
He was completely right as that was how it worked the encryption that Javier sent us.

It is good to have geniouses around ;-)

After adding the decryption to the driver the game looks like this:

It has a few graphic glitches that will be fixed soon but it seems to be PLAYABLE ;-)

Update: more progress has been made. You can read about it here

Javier nos ha mandado (via ElSemi) información detallada sobre cómo funciona el proceso de encriptación. Sin embargo, las hojas donde tenía detallado el funcionamiento del algoritmo eran difíciles de leer en algunos sitios (ya que la protección fue diseñada hace 15 años) por lo que probablemente habría alguna errata en el fichero Excel que nos mandó. ElSemi y yo intentamos en vano añadir la información de Javier. Nicola Salmoria ha realizado un trabajo excelente (como siempre) terminando de pulir esa información y descubrir los detalles que faltaban. El ha implementado una función de desencriptación que funciona no solo para el World Rally, sino también para el Squash y el Thunder Hoop.

Mike Coates terminó el interfaz para obtener datos de la placa del World Rally, suministrando valores encriptados y sus correspondientes valores desencriptados, que permitieron obtener los datos específicos necesarios para la correcta desencriptación del World Rally.

Mención especial a Andreas Naive. La información de alto nivel que Javier nos dio hace unos días ya había sido descubierta por Andreas, que la publicó en su página hace unos meses. Antes de que Javier nos diera la información detallada sobre cómo funcionaba la encriptación, contacté con Andreas Naive sobre el tema. Me dijo que ahora mismo no tenía tiempo disponible para mirarlo. Sin embargo, me dijo que tenía un pálpito:

“Al descifrar cada bloque de 16 bits, se hacía en tres trozos: un primer bloque de 6 bits y luego dos bloques de 5 bits, cada uno de ellos basados en un/unos bits del primer bloque. No recuerdo cuáles eran cuáles, pero el primer bloque era el que tenía una estructura más sencilla (según las tablas que publiqué), mientras que los otros dos son los que mostraban estructura más compleja (y efectos de acarreo debidos a alguna suma).”

El algoritmo que nos había mandado Javier funciona exactamente de esa forma.

Está bien estar rodeado de genios ;-)

Después de añadir el algoritmo para desencriptar los datos, el juego mostraba este aspecto:

El juego tiene algunos fallos gráficos que se corregirán pronto pero es JUGABLE (lo poco que he probado) ;-)

Update: se ha completado la emulación del World Rally. Puedes leer sobre ello aquí

More World Rally news

Manu — Wed, 07 May 2008 06:18:05 GMT

This post is first in English and then in Spanish
Esta noticia está primero en inglés y luego en español

The Dallas DS5002FP core is complete. The additions to the 8051 core are:

- Extra Special Function Registers
- Byte-wide Bus Support
- Memory Partition and Memory Range
- Bootstrap Configuration
- Power Fail Interrupt
- Timed Access
- Stop Mode
- Idle Mode

I didn’t bother to add support for Peripherals, the Reprogrammable Peripheral Controller or CRC-16 support because it isn’t used by the game.

The communication between the Dallas and the main CPU seems to be working, however, the game writes some bytes to the encrypted Video RAM, it reads them back and then makes some calls based on the decrypted data. As the current decryption is wrong, the game doesn’t work yet and never uses the dallas for anything interesting right now.

The decryption of the Video RAM is in the same state as it was. However, ElSemi asked Javier for information about the decryption and he is trying to help us. He didn’t remember how it was performed but we send him the information we got about it in 2002 (when Mike connected the PCB to the fluke) to refresh a bit his memory. Thanks to that, he gave us another point of view about how the encryption may work. Even if he is right, we still need to get data from the original PCB. Hopefully, somewhere in the process we end up with a clean decrypt function that uses no tables.

Understanding some code I was able to guess some decrypted values and get this:

It looks really ugly as the colors for the tiles aren’t decrypted properly but this finally confirms that World Rally uses the same Video Hardware as Target Hits. The only change seems to be the additional palette entries for the night races. I have seen how the palette was configured in a night race and I have an idea of how the palette works for the night races (not very different from World Rally 2), but we’ll have to wait until the game works to implement it.

So, the current status is that if the Video RAM is decrypted properly and we are lucky, the game should work without the light effects (and maybe some priority problems) and be playable. If we are not that lucky, hopefully the problems are not difficult to find and fix.

Update: more progress has been made. You can read about it here

El emulador del Dallas DS5002FP está completado. Lo que se ha añadido al emulador del 8051es:

- Registros de funciones especiales (SFRs) extra
- Soporte del Byte-wide bus
- Partición y Rango de memoria
- Configuration de arranque
- Interrupción por falta de energía
- Acceso temporizado
- Modo Stop
- Modo Idle

No he añadido soporte de periféricos ni el Controlador Reprogramable de Periféricos, porque no son usados por el juego.

La comunicación entre el Dallas y la CPU principal parece estar funcionando correctamente. Sin embargo, el juego escribe algunos bytes a la memoria de Video encriptada, luego los lee desencriptados y hace unos saltos dependiendo del valor desencriptado. Como la desencriptación actual es incorrecta, el juego sigue sin funcionar todavía y no utiliza el dallas para nada interesante todavía.

La desencriptación de la memoria de Video sigue igual. Sin embargo, ElSemi le ha pedido a Javier información sobre la encriptación y nos está tratando de ayudar. Javier no recordaba cómo funcionaba la encriptación por lo que le hemos mandado la información que obtuvimos en 2002 (cuando Mike conectó la placa al fluke) para refrescar su memoria. Gracias a ello, Javier nos ha dado otro punto de vista de cómo podría funcionar la encriptación. Aún si está en lo cierto, con lo que nos ha dicho aún necesitamos obtener datos de la placa original. Con un poco de suerte, en algún momento terminaremos con una función de desencriptación clara que no usa tablas.

Entendiendo un poco de código he podido intuir unos cuantos valores desencriptados y obtener esto:

Se ve realmente feo porque los colores de los bloques no están desencriptados correctamente pero esto confirma finalmente que el World Rally utiliza el mismo hardware gráfico que el Target Hits. El único cambio son unas entradas adicionales de la paleta para las carreras nocturnas. He podido ver como se configuraba la paleta en una carrera nocturna y tengo una idea de cómo va (parecido al World Rally 2), pero habrá que esperar a que el juego funcione para poder implementarlo.

Por lo tanto, el estado actual es que si la RAM de Video se desencripta correctamente y tenemos un poco de suerte, el juego debería funcionar sin los efectos de luz (y quizás con algún problema de prioridades), pero sería jugable. Si no somos tan afortunados, con un poco más de trabajo encontraremos y arreglaremos los problemas que surjan.

Actualización: se ha avanzado un poco más. Puedes leer sobre ello aquí

World Rally, finally

Manu — Mon, 05 May 2008 19:19:39 GMT

This post is first in English and then in Spanish
Esta noticia está primero en inglés y luego en español

If you have tried to execute World Rally since it was added to MAME you have seen the precarious state of the driver. The protection of World Rally is double:

Encrypted Video RAM:

The encrypted Video RAM transforms a 32 bits value written to it to another value that is really complex to derive from the original value. This protection was also used in Thunder Hoop and Squash. Mike Coates connected a fluke to his PCBs and got a 4 GB file will all possible combinations (2^32) for Thunder Hoop and Squash. Using that file, the encryption process was replicated. However, it was not very practical to require a 16 GB file to run the game. Nicola Salmoria studied the file and generated some code (about 400 lines) that replicated the original encryption. That’s why you can play Thunder Hoop and Squash.

Why World Rally encryption is incomplete? All of us wanted to see World Rally running before any other game, so we started studying World Rally before Squash and Thunder Hoop. In that time, we thought that the encryption was only performed in 16 bits, not 32. Unfortunately, when Mike got all decrypted values (2^16), the code was clearly showing that the decrypted values were not correct in most cases (for example, one of the methods that draws the tiles uses a decrypted value for the number of tiles to draw. That’s why the power on self test results and other screens look horrible right now). After we find that out, the decryption efforts were centered in the other games. As World Rally was not going to be playable unless the dallas code was dumped (something that seemed unlikely), it has remained that way.

Dallas DS5002FP secure MCU:

The Dallas MCU in the Gaelco games has some critical code that handles inputs, objects manipulation and other complex calculations that prevent the game from working properly (Take a look at World Rally 2 if you don’t know what I mean). The DS5002FP MCU has been one of the best chips for protecting information on its era making emulation of these games impossible for now. There are some specialized labs that claim to be able to get the code from a DS5002FP but it is an incredibly expensive process we can’t afford.

I have to admit that I thought that I’d never see World Rally emulated, as I approached some people at Gaelco and Zigurat for the Dallas code without luck. However, the other day I received an email that changed my mind. Miguel Angel Horna (ElSemi) did a talk at Lleida Lan Party about emulation. At the party he met one of the Gaelco founders (Javier Valero) that was there to speak about the arcade game industry. ElSemi and Javier talked about the protection in some Gaelco games and Javier was kind enough to provide the dallas code for World Rally. With the internal code of the DS5002FP it is just a matter of time to finally have a working driver in MAME.

I have been completely inactive from MAME development for a while but as I worked on the drivers of all the Gaelco 2D games (except Master Boy) and I spent a lot of time and money on them, I thought it was a good time to resume my work on MAME.

Mike Coates is trying to get all the possible values from World Rally Video RAM so we can advance in that area too.

Currently I am writing a DS5002FP core to run the dallas code. The DS5002FP is just a 8051 MCU with some additions and as MAME already has a 8051 core things are going fast. For example, the initial coprocessor test works (although it is a trivial test to pass without the dallas code):

Also, adding the Power Fail Interrupt of the dallas I managed to see this screen that is supposed to be seen when the dallas is dying:

After the both protections are emulated the driver will need more work as probably the video hardware emulation isn’t complete and needs some tweaks here and there but I’m confident that World Rally will be playable in a not so distant future. I don’t know if Javier will give us more dallas code or not, but hats off to him for this one. Also, thanks a lot to ElSemi for being able to get the dallas code from him.

Update: more progress has been made. You can read about it here

Si has tratado de ejecutar el World Rally desde que fue añadido al MAME habrás visto el estado tan precario del driver. La protección del World Rally es doble:

Memoria de Video Encriptada:

La memoria de video encriptada transforma los valores de 32 bits que se escriben en ella, de forma que el valor resultante es extremadamente difícil de obtenerlo dado el valor original. Esta protección también se utilizó en el Thunder Hoop y en el Squash. Mike Coates conecto un fluke a su placa y obtuvo un archivo de 4 GB con todas las combinaciones posibles (2^32). Usando ese archivo, se pudo replicar la encriptación. Sin embargo, no era muy práctico necesitar un archivo de 16 GB para ejecutar el juego. Nicola Salmoria estudió el archivo y obtuvo un código (cerca de 400 líneas) que obtenía los mismos valores de la encriptación. Por ello se puede jugar al Thunder Hoop y al Squash actualmente.

¿Por qué la encriptación del World Rally está incompleta? Todos queríamos ver el World Rally funcionando antes que cualquier otro juego, por lo que empezamos a estudiar el World Rally antes que el Squash y el Thunder Hoop. En esa época, pensábamos que la encriptación solo era de 16 bits, no de 32. Cuando Mike obtuvo los valores desencriptados (2^16) pudimos comprobar la cruda realidad. El código mostraba claramente que los valores desencriptados no eran correctos en la mayoría de los casos (por ejemplo, uno de los métodos que dibuja los fondos de las pantallas usa un valor desencriptado como el número de bloques a dibujar. Por eso la carta de ajuste inicial que muestra los resultados de los tests iniciales se ve tan mal actualmente). Después de percatarnos de nuestro error, los esfuerzos se centraron en el resto de juegos. Como el World Rally no iba a ser jugable a no ser que el código del dallas se obtuviera (cosa que parecía altamente improbable), se ha mantenido como estaba.

Microcontrollador Dallas DS5002FP:

El Microcontrolador Dallas en los juegos de Gaelco contiene código crítico que maneja la entradas de los jugadores, la manipulación de objetos u otros cálculos complejos que hacen que los juegos no funcionen correctamente (prueba el World Rally 2 si no sabes a lo que me refiero). El DS5002FP ha sido uno de los chips con más éxito a la hora de proteger información en su época, haciendo que la emulación de estos juegos fuera imposible. Hay laboratorios especializados que presumen de poder obtener el código de un DS5002FP pero es un proceso increíblemente caro que no nos podemos permitir.

Debo admitir que pensé que nunca vería el Word Rally emulado, ya que contacté con varias personas de Gaelco y Zigurat preguntando por el código del Dallas pero no hubo suerte en ese momento. Sin embargo, el otro día recibí un email que me hizo cambiar de opinión. Miguel Angel Horna (ElSemi) dio una charla en la Lleida Lan Party sobre emulación. En la party conoció a uno de los fundadores de Gaelco (Javier Valero) que estaba allí para hablar acerca de la industria de los juegos recreativos. ElSemi y Javier hablaron sobre la protección en algunos juegos de Gaelco y Javier tuvo el detalle de dar el código del dallas que usa el World Rally. Con el código interno del DS5002FP es sólo una cuestión de tiempo de que finalmente el juego pueda funcionar correctamente en el MAME.

He estado completamente inactivo en el desarrollo de cosas para el MAME bastante tiempo, pero como trabajé en los drivers de todos los juegos 2D de Gaelco (excepto el Master Boy) y me he dejado un montón de tiempo y dinero en ellos, he pensado que sería un buen momento para trabajar un poco más en el MAME.

Mike Coates está tratando de obtener los posibles valores de la memoria encriptada del World Rally para ir avanzando en ese aspecto.

Por mi parte, estoy escribiendo un core para el procesador DS5002FP de forma que se pueda ejecutar el código del dallas en MAME. El DS5002FP es un microcontrolador 8051 con ligeros cambios, y como ya hay un core del 8051 en MAME, la cosa avanza rápido. Por ejemplo, las comprobaciones iniciales del coprocesador funcionan (aunque es un test tan sencillo, que se puede parchear sin tener el código del dallas, pero bueno):

Emulando la interrupción de fallo de tension del dallas, he conseguido ver esta pantalla que se muestra cuando se le acaba la pila al dallas y su contenido deja de funcionar:

Después de que ámbas protecciones hayan sido emuladas el driver necesitara un poco más de trabajo porque probablemente la emulación de video requiera unas mejoras pero estoy convencido de que el World Rally será jugable en un futuro no muy lejano. No se si Javier nos pasará más códigos del dallas para otros juegos o no, pero me quito el sombrero por lo que ya nos ha dado. Muchísimas gracias a ElSemi por conseguir el código del Dallas.

Actualización: se ha avanzado un poco más. Puedes leer sobre ello aquí

Google and AdSense

Manu — Mon, 28 Apr 2008 21:25:28 GMT

Probably most of you have heard about AdSense, the ad serving program run by Google. It is very simple to have a website to display ads using AdSense, as you only need to include an external script with a few parameters (that mainly configure the size and colors of the ads). For each click in a banner of the website, the associated AdSense account gets a bit of money.

However, this is not the only way to get AdSense in your pages. Google has the "AdSense API", which consists in several web services that allow sharing the revenue of the banners between the website owner and the content publisher. That way, website with user generated content can pay their users if their content is useful.

A customer asked me to modify their website so they can share some of the benefits of their banners with the creators of the articles in their site using the AdSense API. I’ll detail some of my experiences with the integration of the AdSense API in the webpage.

As I had to deal with web services, and I was comfortable with the web service support present in the .NET Framework 2.0, the easiest thing to do for me was to follow that route. However, I always try to learn new stuff when I can, so it was an excellent chance for me to learn WCF. Creating a proxy for a web service in WCF is as easy as in .NET Framework 2.0. However, the problems started really soon. For some unknown reason, the Google guys require some SOAP headers in the web service calls that are not exposed in the AdSense’s WSDL. Reading some of the comments in the AdSense Group revealed that the SOAP headers were exposed previously but they were removed without giving any reason. I found this to be really annoying and. It can be fixed adding the headers to the OutgoingMessageHeaders in the current OperationScope. Only for this detail the Adsense API can be labeled as unfriendly for the developers.

Once you set the headers manually, it is easy to call to the different methods to create AdSense accounts, associate them with a publisher and generate the code to display ads. However, I found very confusing some parts of the documentation, specially all the ids available (you end up using clientId, publisherId, synServiceId, developerId, and in some methods having to pass the same id twice).

Inspecting the generated code for the WCF proxy, I found one thing I didn’t like about WCF. The DataContractSerializer does not handle bare arrays, so the generated code was using the XmlSerializer for a lot of things that need directly or indirectly to handle bare arrays in the AdSense API. The official statement of Microsoft about this seems to be that bare arrays do not support the distinction between null arrays and empty arrays, so the best is to use a wrapped collection.

The AdSense API has a technical requirements page that informs you how to proceed in some interactions with the users and how to react to error conditions. The AdSense API methods expose the fault details as specified in the AdSenseApiException. When an error happens when your WCF proxy is calling to the AdSense API, a FaultException is generated. I was expecting the proxy to receive a FaultException< AdSenseApiException> instead so I could easily access to the exception details but that wasn’t the case (maybe that only works in WCF to WCF scenarios?). I tried to create a generic error handler in order to convert the SOAP faults to the FaultException<AdSenseApiException> type, but the information I found about WCF error handling in the web was only talking about server side error handling using the IErrorHandler interface. It was clear that I needed more in depth knowledge of WCF in order to do that. After some research I bought Inside Windows Communication Foundation:

The book was exactly what I was looking for. Good and detailed internal information about how WCF worked.

Thanks to the book I was able to make my proxy generate the FaultException<AdSenseApiException> transparently, but it wasn’t easy. Well, it is not difficult but you have to create a lot of classes to do something simple.

A channel exposes a method called GetProperty<T>. When an exception is generated in the channel, the GetProperty<T> method is called with a parameter of type FaultConverter. The method should return an an instance of a FaultConverter, which creates the appropriate exception/fault type. To solve my problem I created an AdSenseFaultConverter:

However, in order to expose the AdSenseFaultConverter I had to create a ChannelFactory (AdSenseChannelFactory), a channel (AdSenseChannelBase and AdSenseRequestChannel), a Binding (AdSenseHttpBinding), and a BindingElement (AdSenseBindingElement). A lot of classes with little added value just to override the GetProperty<T> method in the channel. Also, to be able to use the configuration instead of setting up the communication programmatically, I had to create a configuration BindingElement(AdSenseHttpBindingElement) and a CollectionBindingElement (AdSenseHttpBindingCollectionElement). As I said before, it is a lot of code.

One thing that I haven’t commented yet is that your AdSense API implementation needs to be reviewed in order to be able to serve live ads, and you need to follow some policy requirements that you need to carefully study as they want to control even the help pages you put in your web site. One of the requirements to be eligible to participate in the AdSense API program is to have more than 100000 daily page views. This is a showstopper for most of the web sites making the AdSense API useless for 99.9% of the people.

Another negative point of the AdSense API is that the WSDL has been broken for more than a month. I planned to write this article a couple of months ago so I don’t know if the WSDL of the AdSense API is still invalid or not, but having it broken for more than a month can give you an idea of the overall satisfaction I have with the AdSense API.

To finish talking about the AdSense API, I have to mention the Sandbox. The development web services don’t behave exactly as the live web services (something that I hate), so some methods need to have additional headers when testing the API to work properly. Why? Well, basically because they do not have a decent sandbox. They just have the development version of the web services and you have to live with that. No GUI for managing the developers accounts, payments or something similar to the AdSense account webpage. If you compare this to other sandboxes like the paypal one where you can even receive the IPN confirmation messages you can clearly see the difference.

And the last rant of the post is about Google policies regarding to AdWords and AdSense. If you are an European citizen and you want to advertise using AdWords, Google forces you to pay in Euros. However, when you have AdSense in your page, Google will pay you in dollars. Judging by the prices of the ads in AdWords and the money you receive with AdSense, Google is taking advantage of the European citizens by charging a 60% more for an ad in AdWords and paying a 60% less for it (the exchange rate is approximately 1 euro = 1.60 dollars). Is the European commission too busy fining Microsoft for including free products with Windows to investigate things like this?

To sum up, I’m completely disappointed with Google and their AdSense API. If some people ever had the illusion that Google was a non evil company that did the things the right way, they need to wake up.

NDepend

Manu — Fri, 08 Feb 2008 12:22:13 GMT

One of the tools I’m using is NDepend (http://www.ndepend.com/). It is a cool application that can be used for a lot of things, as will be detailed later.

To use NDepend you have to feed it with a set of assemblies. After that, it will analyze those assemblies and their reference assemblies. If the PDBs of the assemblies are available it will use them so more data is analyzed, although the PDBs are not needed for NDepend to work.

After the analysis has been completed, NDepend looks like this:

I have used version 1.1 of Spring.NET Framework (http://www.springframework.net/) for the sample. As you can see the application has a beautiful GUI. I’ll talk a bit about the main windows of the tool:

• Metrics: this window gives a graphical high level view of the project. You can see a lot of ellipses and some of them that are made of smaller ellipses. The big ellipses are types of the project. If a big ellipse is made of several smaller ellipses, those smaller ellipses are the methods of the type. The types of the same namespace are distributed in the same rectangular region, separated by a small yellow line. The assemblies are separated by a thicker yellow line.

By watching this window you can get an idea of which assemblies contain more code, which types are big, which types have a lot of methods, etc. The window was in “Method Level”. The available levels are Assembly, Namespace, Type, Method and Field.

• Class Browser: this is where the assemblies of the project and its referenced assemblies are shown. You can expand the assembly up to the field level. While you hover over an item in most windows, the Metrics window gets updated highlighting the hovered item.

What it’s really cool about the Class Browser is that for the referenced assemblies, only the namespaces, types, methods and other items that are used by the project are shown.

• Info: When you hover over an item the information about it is displayed here, with some metrics if applicable. There are a lot of metrics so it can take time to interiorize them and judge their value appropriately.

• Dependencies: this window is a bit intimidating at first but like a lot of things, with a bit of effort you can master it. The window displays very important information of the dependencies in a matrix form. As in other windows, the detail level is selectable, from the assembly level to the field level.

A cell can have 3 colors. Blue means that the associated item in the top header uses the associated item in the left header. Green means that the associated item in the left header is using the associated item in the top header. Black means a cyclic dependency. The blue and green cells express the same dependency using opposite points of view.

You can see a diagram of the dependencies clicking on a cell. For example, if you click in the cell corresponding to System.Web.Extensions and System.Web you get this:

To read more about the theory of dependencies take a look here:

ttp://www.theserverside.net/tt/articles/showarticle.tss?id=ControllingDependencies

• CQL Queries: CQL stands for Code Query Language and it is a central piece of NDepend. Being able to ask questions about the code is great. If you like using the Analyzer window in Reflector you’ll love this. You can create new queries and modify the existing ones.

I have presented the main NDepend windows, but the main question is: what you can do with NDepend?

There are several uses that I can think of:

• It is an excellent tool to check differences between different builds of an assembly. You can use it as a diff tool, as it will show in the Class Browser which types, methods, etc have been added, removed or changed. This information is also available for CQL Queries, with the power that it implies.

• It can be used to understand code. When you are thrown with code that you have no clue about how it works you have several options to accomplish that task:

Look at the source code directly.
Make some diagrams from the source code (for example, using View Class Diagram in Visual Studio).
Use NDepend.

These methods are not exclusive, and probably you’ll use a combination of them to understand how the code works. However, the abstraction level is higher when you use NDepend is higher than when you use a Class Diagram (and a Class Diagram is a high level view compared to raw source code). I’d use NDepend to have a global vision of the main assemblies, type most used, dependencies between namespaces, more complex methods, etc. and then focus my attention in the most important parts of the application, going to a lower level of detail.

• It can be used in the refactoring process, to see the impact of a change, and to keep dependencies to a minimum, in order to have a project that is easier to understand and maintain.

• Integrating NDepend in the software development process. IMHO this is a key point, as using NDepend regulary will certainly improve the quality of the software that is being developed. NDepend can generate a report of a project. A sample report is available here:

http://www.ndepend.com/SampleReports/OnNUnit/NDependReport.html

There are a lot of sections in the report, but I’ll concentrate on the section called “CQL Queries and Constraints”. That section is really important as it shows a list of items that can be problematic or don’t follow the standards of your company (methods with a lot of lines of code, very complex methods, poorly commented methods, types or methods or properties or fields with incorrect naming conventions, etc). This way the project leader can make periodic checks of those problems and report them to the people that are responsible for them.
This also allows you to know more the people you’re working with. Some developers don’t comment code, others like to make very long methods, and others tend to write complex logic that is difficult to understand by others. With the review of the report, they can be instructed so the resulting code and comments is better and more uniform.

However, there are times where a method with more than 30 lines of code is necessary. Or a method with a cyclomatic complexity of 25 is needed. So it is up to the project leader to review the suspicious method and inform the developers of the problem or simply acknowledge that it is necessary.

Unfortunately, NDepend does not provide built in support for this level of detail of project management. I have been told that in a future version it will provide hooks so custom scenarios like the exposed above are fully supported.

So, after what I have said about NDepend, you can be thinking, is it for me? Well, that depends on a lot of factors. I know a lot of companies that have no interest in achieving high quality products and that they only care about having a project done as soon as possible. If the maintenance of that project is a nightmare, well, that is another problem for another moment. However, there are some companies that actually spend time and effort to produce excellent products, so they may be interested in what NDepend can add for them.

I have used NDepend in several projects that are widely used when learning about it, and I found something curious:

The image shows that almost all of the NHibernate code has a big dependency cycle. I wish them good luck making the transition to ASTs.

To finish this post, I have to say that I have received a free professional version of NDepend. This has not conditioned my post about it. If I think it was a useless product I will not have posted about it.