Technical Taco

Building a Metal Gear Mk.II-Inspired Self-Balancing Robot (Part 4: Traction and Balance)

Fri, 17 Apr 2026 00:00:00 +0000

Iterating on a Tire Design

The game designers for the Mk.II added a distinctive tread pattern to the wheels, and to me it looked both functional and visually appealing. The tread’s angular pattern resembles a circuit board which is quite fitting to the personality of the bot.

In order to replicate this pattern, I had to make use of the 3rd-party Curves Workbench in FreeCAD. This feature projects curved surfaces onto a flat sketch, letting you use familiar tools and constraints. Surprisingly, this turned out to be one of the easiest tasks on the bot, thanks to the simplicity of the workbench.

The first version of the tire was printed with basic slicer settings in TPU. I expected TPU’s softness to aid in traction, but I immediately learned that it was simply too smooth to get anywhere. There was, however, one slicer trick I knew I could try: fuzzy walls.

The first of my working prototypes used these fuzzy-walled tires. While they absolutely improved performance, they still did not have very much traction on hard surfaces (even slightly rough ones). Only on carpet did this iteration have anywhere near enough traction to balance its weight.

My friend Arthur, who printed the whole first prototype for me, shared a trick he used in combat robotics: coat the wheels with latex paint to give them extra traction. This worked, but it wore off far too quickly. Arthur also encouraged me to join him at a couple of his robotics competitions, and there I met someone who shared a solution to this problem using urethane rubber casting.

Molding

From the combat robotics meetup, I learned that designing a tire mold was pretty easy: subtract the tire from a solid block and slice it into 4 parts. I grabbed some 30A Hardness urethane and, due to the complex tread pattern, I chose to purchase some mold release as well.

I secured the wheels into the center of the molds, screwed them down, and poured the rubber.

The final iteration of the wheels came out great. The sprues (extra bits) were easily cut off by passing a hobby knife between the lid and the rest of the mold. The rubber tires have performed well on a variety of surfaces, both soft and hard. With traction solved, the next challenge was stability.

Balance Logic

Getting a bot to balance is theoretically straightforward. Similar to a car in cruise control, it’s largely a problem of controlling a single variable. In the case of the car, the variable would be the speed; for the Mk.II, it is the pitch angle. PID controllers are a very common way to do this since they model many types of problems reasonably well. PID controllers relate the difference in the target variable to some other output, such as throttle or torque. Their robustness comes from their use of the “present” (the Proportional term), the “past” (the Integral term), and the “future” (the Derivative term).

In practice, optimizing only a single variable won’t work without precise calibration. So, we have to control an additional variable: the target pitch angle. If our robot drifts to one side or another, we can use a second PID controller to adjust the target pitch in the opposite direction.

This turns out to be sufficient to get the bot to balance. Here are some resources to learn more about this specific arrangement and PID tuning in general:

Dual Board Control

Having seen good results with the small-scale prototype, I began putting together the full-sized bot. Still, the testing showed some issues that needed to be addressed. The ESP32-S3 only supports BLE (Bluetooth Low Energy) but all of the controllers I wanted to use needed Bluetooth Classic. In addition, the IMU I had purchased gave incredibly noisy data but only intermittently.

I purchased an ESP32 DevKit v1 and tested it with the small scale prototype. While it looked like it would have the specs needed to run the bot, it turned out to be very difficult to get a 1khz control loop to run smoothly. This seemed to put me in a worse position at first, but I realized that I had written-off a perfectly good solution: using two boards together.

Using both the STM32 and ESP32 allowed me to leverage their individual strengths. My STM32 Discovery has an onboard IMU and the performance needed to run the balance control loop. The ESP32 can asynchronously communicate with many clients at once: Bluetooth, WiFi, UART, etc. I chose to use lwpkt on top of UART to frame messages between the boards and to reject bad packets.


language: mermaid
flowchart LR
  ds3
  xinput
  subgraph bot
    esp32[ESP32 DEVKITV1]
    stm32[STM32F411E-DISCO]
    esp32 --UART (Input & Console)--> stm32
    stm32 --UART (Metrics & Logs)--> esp32
  end
  xinput --USB (Input)--> linq
  ds3 --Bluetooth (Input)--> esp32
  linq --UDP:2008 (Input)--> esp32
  linq --UDP:2014 (Console)--> esp32
  esp32 --UDP:2008 (Metrics)--> telegraf
  esp32 --UDP:2014 (Logs)--> telegraf
  subgraph pc
    linq[C# Desktop App]
    grafana[Grafana]
    influx[(InfluxDB)]
    telegraf[Telegraf]
    grafana <--HTTP:8086--> influx
    linq <--HTTP:8086--> influx
    telegraf --HTTP:8086--> influx
  end

Assembly

With the decision to use two microcontroller boards came the requirement to mount them securely. I designed a set of lightweigth brackets to hold the boards together.

The Mk.II model in game has a back-panel that resembles a 1990’s camcorder. On the back, it says “整備ケーブルコネクタ口ケーブルが挿す上下が気を付けろ” which Google tells me means “Maintenance Cable Port: Pay Attention to Connector Orientation”. While I’m a stickler for details, I decided that USB-C Panel Mount connectors would be acceptable. These were connected as the boards were installed with the help of a couple of adapters:

I also added an external USB-B cable to represent the Mk.II’s “tentacle.” In the game, he uses it to manipulate both objects and computers. While I don’t yet have the ability to make a prehensile arm, I was able to give him mild hacking abilities.

Phantom Vibration

Once fully assembled, however, the Mk.II developed an odd vibration. I first suspected this to be caused by excessively high PID coefficients.

However, no amount of tuning could remove the oscillation. After chatting with the folks in the SimpleFOC discord again, I learned that the actual source of the issue was the placement of the accelerometer away from the center of rotation. My decision to pivot to the STM32 Discovery board placed the IMU several centimeters above the center of mass.

@copper280z told me that, in a configuration like this, the accelerometer will feel forces from rotation which it cannot distinguish from gravity. While this made sense, I was intrigued to learn that the effect can be subtracted out using the gyroscope if the exact distance from the center of rotation is known.

At the time, I did not have the gyroscope working. I had to debug the Adafruit library to add support for the onboard IMU. If you see the Adafruit maintainers, kindly ask them to look at my pull request.

I was able to reduce the vibrations to a small jitter by estimating the offset and subtracting out a centripetal term. It seemed that fully eliminating the effect would again require precise calibration. I chatted with Art from robomates about how he solved this problem for his bots. He said that his DMP IMU solved it for him.

After a bit of searching and a bit more chatting with @copper280z I finally learned what to do. The final trick to get rid of the vibration was to use the gyroscope as the primary source of pitch information. This can only happen after finding an absolute “down” direction with the accelerometer. I implemented this for the Mk.II using a complementary filter weighted 98% to the gyroscope. This completely eliminated the vibration while still allowing the accelerometer to provide long-term stability.

Coming Up…

In the final post, I’ll show how to control the Mk.II in real time, along with the telemetry and tooling I use to debug it while it’s running.

Previous Post: Part 3: CAD Design

Building a Metal Gear Mk.II-Inspired Self-Balancing Robot (Part 3: CAD Design)

Sun, 12 Apr 2026 00:00:00 +0000

FreeCAD

When I started the Mk.II project, I didn’t have any meaningful CAD experience. I had been watching channels like Maker’s Muse, Wintergatan, BPS.space, and rctestflight do amazing things with CAD, so I had some idea of what to do. I chose to learn FreeCAD because of its open-source nature and fully-featured user experience. In order to get up to speed, I made use of the FreeCAD 101 series on YouTube as well as Mangojelly’s videos:

These tutorials helped me learn the UI idiosyncrasies and the specific features I needed in order to model the Mk.II. I was able to recreate a relatively accurate model of him over the course of a couple of months.

This is not to say that the journey was easy. FreeCAD has two really big issues that cause user friction:

Edge/face renames break history & sketches
B-splines get introduced for linear features by some operations.

These issues will absolutely break your workflow if you don’t plan around them. I have managed to find workarounds that avoid these issues wherever possible.

Dodging FreeCAD Roadblocks

Master Sketches

The first strategy that I want to share is the use of Master Sketches. FreeCAD’s operations, like Pad, Pocket, Groove, etc. operate on the selection they are given. To put it another way, you can select part of a sketch in order to limit the operation to that part. For example, take this simple shape:

The most straight-forward way to construct this is by Padding (extruding) with a sketch drawn on the top face, like this:

This works well if you are using absolute measurements everywhere. However, as soon as you need constraints between e.g. the larger block and smaller one, then this strategy would have you depending on specific edges and faces of the output geometry. Output geometry can and will change as you are iterating on a design, so this is a shaky foundation. You will end up breaking the references that you constrained your sketch to.

A better solution is to put as many of the shapes and constraints into a single sketch as you can. Then, only one sketch drives the outputs of your object, and each feature is independent of the output geometry of the others.

Note that both Pad operations in this example refer to the same sketch. Destructive operations such as deleting sketch edges will still potentially lead to broken geometry, but all additive modifications to the sketch will apply without breaking any unrelated geometry. This turns out to be a specific form of my next strategy.

Refer to the earliest geometry possible.

The main benefit of Master Sketches is that they automatically force your operations to refer to the earliest possible feature (the Master Sketch itself). Since none of your operations rely on the outputs of any others, they cannot break each other’s constraints. This can be generalized to situations where a single sketch is not sufficient.

If you must base a feature or sketch on output geometry, reference the earliest instance of that geometry whenever possible. In particular, avoid referencing the output of chamfer and fillet operations, preferring to reference the unmodified faces from the previous feature tree item.

Use Clones and the Part Workbench

Cloning or shape-binding a Master Object is another way to ensure your feature tree stays linear and short. If you have two parts that interface in some way, creating a single Master Object and deriving both parts from that will prevent your tree from getting tangled.

In addition, several operations in the Part Design workbench do not allow access to modify earlier features in the feature tree. Namely the boolean operations and the Hole operation as they “consume” the previous node in the feature tree. There are two strategies to avoid getting stuck, unable to modify those past objects.

The first option is to always clone your object before performing a boolean operation in the Part Design workbench. The clone will track changes to the “tip” of the object it references, allowing you to make changes to that node that propagate.

The second option is to use the Part workbench to do the boolean operation and then converting the object back into a Part Design body. This option is my preferred way of working as it keeps the feature tree a bit more tidy.

Having both of these options available has helped me work around a few bugs, so it’s good to keep both in mind.

Don’t Reinvent the GM4108H-120T

I purchased several off-the-shelf components for my bot. While I might eventually design my own board, for the prototypes I chose to use single-purpose module boards and dev kit boards. These have the advantage of being well documented. This usually means that there are CAD files with the exact part dimensions already available online. The motors I chose are also a very standard entity. Because they are being used for drones and other robotics projects, the CAD files for them have been released as well.

Several of the components I purchased didn’t have existing CAD models. Rather than design them into the Mk.II, I chose to design separate models and import them:

I used these reference models as Master Objects via clones and shape-binders to drive the interfaces between them and the bot. But just using the parts alone was not enough to guarantee things fit.

Designing for 3D-Printing

Parts Have Layers, Like an Ogre

3D printers typically produce objects with horizontal layers. There are, of course, exceptions including belt-fed printers and unconstrained toolpath generation (like https://www.gerridaj.com/ or https://fullcontrol.xyz). But even in these cases 3D prints tend to have weakpoints due to layer adhesion.

These implicit weakpoints necessitated design consideration. For each of my parts I applied this simple process to work around these weaknesses:

Determine the limits of stress that the part will experience. Make particular note of tension and shear stresses. Layer adhesion is typically not a problem for compressive stresses.
Plan the layer lines so that the solid filament lines are parallel to the stresses.
Incorporate a large, flat area on your part parallel to your layer lines. You will print with this area on the bed, minimizing support.
- If there is no flat area, plan to slice the part in half and join them. I primarily used screws as fasteners, but also included a couple of snap-fit interfaces.
- If multiple flat areas are available, choose the one that will result in the most visually appealing layer lines, followed by the one that uses the least support material.
Proceed to design your part, applying clearances according to the accuracy of your printer in each dimension.

During the export of any 3D model to FDM G-code, curved edges will get turned into small line segments. This means that any curved surface will have a slightly smaller radius when printed. Also, since 3D printers cannot draw lines significantly narrower than the nozzle width, corners are always rounded.

To compensate for this, clearances need to be added on faces and corners where parts fit together. For my X & Y axes, I used a quarter of my nozzle size (0.1mm) on each face. When adding up the left and right sides, this means I am allowing a total deviation of one-half of a nozzle.

Zero clearance will almost never work, even for snug fit parts. Only the strongest parts can withstand assembly with no clearance.

One thing I learned late in the process was the use of variables. I wish I had introduced a variable for my clearance early on in my project, so please learn from my mistake and use a Variable Set for your printer tolerances, etc.

Watch Your First Layer

The first layer of any 3D print is the most critical. If the filament doesn’t stick to the print bed, your part will not take shape. In contrast, if your first layer prints perfectly, the rest of your print is very likely to succeed (overhangs & bridges notwithstanding). So, the advice you will hear time and again is to always watch the first layer. I can attest that this is the best way to avoid wasting time and material on a print that is doomed to fail.

Coming Up…

If you read up to this point, you’ve seen the process I used to design the Mk.II’s various components. Coming up is where the rubber meets the road (literaly!). We’ll be talking traction, balance, PID tuning and finally showing off the bot!

Previous Post: Part 2: Prototypes Next Post: Part 4: Traction and Balance

Building a Metal Gear Mk.II-Inspired Self-Balancing Robot (Part 2: Prototypes)

Thu, 19 Mar 2026 00:00:00 +0000

In the last post I went over my inspiration and goals for the Mk.II project. This post covers the steps I took to get the various prototypes up and running. I will be including sponsored links to the parts I purchased; if you use these links it will help me continue making content like this!

Parts

At the end of August, after getting my plan reviewed by the experts, I eagerly ordered my first batch of parts.


language: mermaid
graph TB
  battery[4S LIPO]
  splitter[XT90 Splitter]
  step_down[5V 5A step-down]
  rail_14v_in@{ shape: junction }
  rail_5v_in@{ shape: junction }
  battery -- 14.8V --> splitter
  splitter -- 14.8V --> rail_14v_in
  splitter -- 14.8V --> step_down
  step_down -- 5V --> rail_5v_in
  high_level_controller[Pi 5 / Jetson Orin]
  camera
  screen
  low_level_controller[STM32F411E-DISCOVERY]
  motor_controller_left[SimpleFOC Mini]
  motor_controller_right[SimpleFOC Mini]
  motor_left[GM4108 BLDC]
  motor_right[GM4108 BLDC]
  sensor_left[AS5047P]
  sensor_right[AS5047P]
  rail_5v_1@{ shape: junction } -- 5V --> high_level_controller
  rail_5v_2@{ shape: junction } -- 5V --> low_level_controller
  high_level_controller -- Direction --> low_level_controller
  camera -- A/V --> high_level_controller
  high_level_controller -- UI --> screen
  low_level_controller -- Telemetry --> high_level_controller
  low_level_controller -- PWM --> motor_controller_left
  rail_14v_1@{ shape: junction } -- 14.8V --> motor_controller_left
  motor_controller_left --> motor_left
  motor_left --> sensor_left
  rail_5v_3@{ shape: junction } -- 5V --> sensor_left
  sensor_left -- ABZ --> low_level_controller
  low_level_controller -- PWM --> motor_controller_right
  rail_14v_2@{ shape: junction } -- 14.8V --> motor_controller_right
  motor_controller_right --> motor_right
  motor_right --> sensor_right
  rail_5v_4@{ shape: junction } -- 5V --> sensor_right
  sensor_right -- ABZ --> low_level_controller

I sent in orders for the MCU, the motors, sensors, motor controllers, and a few other things:

Boards
- STM32F411E-DISCO
- GM4108H-120T
- 2x SimpleFOC Mini
- 2x AS5047P Magnetic Encoder
- ~~5V Regulator~~ not used in current design
Hardware
- M3 Screws & Nuts (black)
- M3 Screws & Nuts (silver)
Power

Motion Sensing

While I waited for parts to arrive, I began modelling the Mk.II in FreeCAD. I learned quite a bit here even beyond the prototyping phase, and it deserves a post of its own. Stay tuned for more on that in a later installment.

As soon as I got the STM32 board, it was time to get some coding done. I surveyed the embedded development landscape for operating systems and IDEs. What I found was that Platform.IO is pretty much the standard for embedded software, enabling a single project to target multiple boards.

I created a project but couldn’t find Arduino support for my board. I did find support for the STM32F407-DISCO board which was the previous hardware revision. I guessed they were compatible, so I moved forward with that as a project starting point. It didn’t take long before issues started to crop up, likely due to the different MCU settings between the boards. I felt I had to pivot to ZephyrOS on the STM32F411-DISCO, and luckily Platform.IO made this relatively painless:

Before:

[env:disco_f411ve]
platform = ststm32
board = disco_f407vg ;disco_f411ve
framework = arduino

After:

[env:disco_f411ve]
platform = ststm32
board = disco_f411ve
framework = zephyr

The first prototype I wanted to create was something that could display the state of the accelerometer. This was very straight-forward in Zephyr, taking only a few minutes to implement:

static const struct device *const accelerometer = DEVICE_DT_GET_ONE(st_lis2dh);
...
IMUReading reading;
sensor_sample_fetch(accelerometer);
reading.accelerometer[0] = sensor_value_to_double(&val[0]);
reading.accelerometer[1] = sensor_value_to_double(&val[1]);
reading.accelerometer[2] = sensor_value_to_double(&val[2]);

Combined with a basic led_service, this was enough to build a “this-way-up” demo:

Next, I wanted to implement something similar for the gyroscope. Only one problem: Zephyr didn’t know about the STM32’s gyroscope. It took me about an hour to figure out how to inform Zephyr about the device. I was able to reuse most of the details from the STM32F407 board configuration.

&spi1 {
    pinctrl-0 = <&spi1_nss_pa4 &spi1_sck_pa5
                &spi1_miso_pa6 &spi1_mosi_pa7>;
    pinctrl-names = "default";
    status = "okay";
    cs-gpios = <&gpioe 3 GPIO_ACTIVE_LOW>;
    i3g4250d@0 {
        compatible = "st,i3g4250d";
        reg = <0>;
        spi-max-frequency = <1000000>;
    };
};

I sent a PR so that others could benefit and then set my focus on integrating the magnetic sensors. I added an angle_service to the project that was responsible for the AS5047P sensors. The sensors support several modes: ABZ, PWM, UVW, and I²C. If you’ll recall, in the last post I received advice from the SimpleFOC forums that ABZ (aka ABI or Quadrature) was the lowest overhead. Adding this sensor to my Zephyr project as a quadrature decoder was also quite easy:

&timers3 {
    status = "okay";

    qdec: qdec {
        status = "okay";
        pinctrl-0 = <&tim3_ch1_pb4 &tim3_ch2_pb5>;
        pinctrl-names = "default";
        st,counts-per-revolution = <1000>;
    };
};

Using a cheap fridge magnet, I was able to show that the sensor was working:

A couple of times I managed to short out the system by touching the sensor power leads with the magnet’s metal housing. I didn’t realize it at the time, but this was a clear indication that my electronics hygiene needed to improve.

Motion Control

In order to wire up a full motion-control prototype, I ordered a few more things:

24AWG assorted colors
DuPont Connectors
Key Switches
5A Slow Blow Fuse
Glass/Nylon Bearings
~~10k Resistors~~ get an assorted set first, replace with specific values
- Assorted Resistors

My first soldering project was the pair of SimpleFOC minis. The purpose of these little boards is to translate low-voltage signals (3.3v pulses) into higher voltages (~15V) sent to the motor leads.

If you have any soldering experience, you can probably tell there are a few things wrong:

Several connections have too much solder and have bulged up.
One of the joints appears cold. (nFT on the right board)
Flux residue not cleaned off.

Don’t worry too much about this, as I ended up rewiring this board directly and these errors didn’t impact the prototyping progress.

From the software side things weren’t simple. On one hand, I had fully working sensors under ZephyrOS, but on the other hand, I had a proof of concept in the way of the Arduino SimpleFOC Balancer project. At the time, there was only a single lonesome Zephyr project doing FOC: Spinner. I had to decide immediately whether I would attempt to fill the feature gaps to enable a SimpleFOC-like API on Zephyr or to add Arduino support for my board.

I decided, since the process to add a new STM32 Arduino variant was well documented, that adding the board aligned better with my skillset. While this did set me back several days and a couple of dollars for a CP2101 USB UART serial adapter, I do think that my contribution here has already been beneficial to the community. I wasn’t able to convince my local copy of Platform.IO to use my custom build of STM32Duino, so unfortunately I had to avoid Platform.IO for anything except serial debugging from here out.

Nevertheless, after adding a motor_service I was able to get a basic SimpleFOC open-loop motion demo running:

The periodic pausing you see there is due to one out of three phases conflicting with other onboard devices. After switching the signal PWM channels to unused pins, I got smooth open-loop control working:

The Missing Magnet

I purchased some Gold Filament for the prototype. I always equip Gold in Kojima’s Games and wanted to pay homage.

My friend Arthur was kind enough to 3D-print a few of my designs, so I was now able to integrate all of the leg parts into a single package.

I powered up the experiment, saw the motors move, but the angle sensors read zero. I was clearly missing something. I once again implored the help of the SimpleFOC Discord channel. I had assumed that the motors’ own magnetic fields would be captured by the sensor. The AS5047P magnetic sensor does after offer the “UVW” mode, which I understood to be reading the field strength of each motor’s coils. What I did not realize is that this is merely an emulation mode offered by the chip. The chip itself requires a magnet positioned directly above it to work. And it can’t just be any old magnet either; it has to be a diametrically magnetized magnet.

If I wanted to stay true to the Mk.II/III design, I had to mount the inner wheel covers through the motor shaft. The gimbal motors I’m using have room for that, but a magnet would just block the shaft. This meant that, suddenly, I had a very very unique magnet requirement:

Ring magnet
Diametrically magnetized
Small
- < ~3 mm thick
- < ~10 mm outer diameter
As wide of an inner diameter as possible

I searched through every magnet website around but I could not find anything that fit perfectly within my original design. But, as luck would have it, I did find exactly ONE option that was close. It had a much smaller inner diameter than I like, but I wagered: with a small redesign and a change in material, I could make it work.

Once they arrived and were installed, I had every individual component working.

The Hygiene Lesson

I was still depending on my friend Arthur for 3D prints, so I figured the easiest way to get a fully integrated prototype assembled would be to build the body out of cardboard. I grabbed a cardboard box, measured up and cut holes for mounting the ankles, and threw it all together:

I arranged a meet-up to get the 3D printed wheels & tires at our local makerspace. And it was there that the poor electronics hygiene caught up with me. I assembled the wheels, powered the bot on, and… a twitch and then it almost immediately fell silent.

I pulled the power and hooked up USB serial to assess the damage. The board gave me random text in return. I fried the brains and the sensors. Oof. Most likely what happened is that one of the 15V leads from the step-down must have contacted one of the lower-voltage data lines on the STM32. There was nowhere to secure the voltage regulator inside the box, and I had not protected the leads.

At this point, I decided I had to make some specific changes. Going forward, I decided:

All power leads must be sleeved.
Any unshielded male power connector must not have power when disconnected.
- Follow standards for XT connectors.
All power leads must be secured to the chassis.

I designed an enclosure with plenty of tie-down points and asked for Arthur to print it. I also reordered the fried parts: 1 STM32 board and 2 AS5047P sensors (all of my low-voltage devices). It didn’t take long for me to reconsider my choice of STM32. I wanted the bot to be controlled like it is in the game, and this meant using a Bluetooth controller. In parallel, I ordered a few extra parts that could be here quicker.

~~ESP32-S3-DevKitC-1~~ BLE only, not used in latest design
WiFi Antenna
INA219 Power Monitor
3.3V Regulator
ICM-42688 IMU

This further hints at some design changes that we will see in later posts.

Coming Up…

In the upcoming posts, I’ll be covering my CAD journey, traction, PID tuning, designing for 3D printing, and more!

Stay tuned.

Previous Post: Part 1: System Design Next Post: Part 3: CAD Design

Building a Metal Gear Mk.II-Inspired Self-Balancing Robot (Part 1: System Design)

Wed, 11 Mar 2026 00:00:00 +0000

Inspiration

At the end of July last year I happened across a Reddit post showing off the performance of a two-wheeled balancer robot. At the time I was still playing through Metal Gear Solid 4, so I immediately noticed the similarities between their robots and the Metal Gear Mk.II:

I eagerly joined their subreddit and Discord channel to see if their bots were customizable. In the game, the Mk.II (and later the Mk.III) is a friendly companion that the player can pilot. He’s nimble on his wheels and can act as your eyes and hands. This is a perfect fit to Robomate’s bring-your-own controller approach. However, their kits weren’t available yet.

I was too eager to wait, so I started planning my own bot. After a bit of searching, I found a couple of preexisting projects that seemed like good places to start. The most similar project to Mk.II was MABEL, with its independent legs. However, SimpleFOCBalancer won me over by its simplicity as a starting point.

Goals

My goals with this project were to (A) learn basic robotics and (B) achieve as close of a representation of Mk.II as I could manage. Since I had zero practical experience with robotics and a pretty abysmal electronics track record, I knew I needed to take it slow. Mistakes would be made.

I paved this basic roadmap:

IMU + Motor prototype
Simple 2-wheel prototype
Full-size 2-wheel bot
Independent leg movement
Computer Vision integration

#4 and #5 on this list are long-term goals with the project. A bot similar to the Robomates implementation would be wonderful, but even achieving #1 would be a win.

To save costs, I hoped to reuse as many of the prototype components in the final bot as possible. So, based mostly on the Simple FOC Arduino Balancer project, I drafted this design:


language: mermaid
graph TB
  battery[4S LIPO]
  splitter[XT90 Splitter]
  step_down[5V 5A Step-Down]
  rail_14v_in@{ shape: junction }
  rail_5v_in@{ shape: junction }
  battery -- 14.8V --> splitter
  splitter -- 14.8V --> rail_14v_in
  splitter -- 14.8V --> step_down
  step_down -- 5V --> rail_5v_in
  high_level_controller[Pi 5 / Jetson Orin]
  camera
  screen
  low_level_controller[STM32F3DISCOVERY]
  motor_controller_left[SimpleFOC Mini]
  motor_controller_right[SimpleFOC Mini]
  motor_left[2408 BLDC]
  motor_right[2408 BLDC]
  sensor_left[AS5600]
  sensor_right[AS5600]
  rail_5v_1@{ shape: junction } -- 5V --> high_level_controller
  rail_5v_2@{ shape: junction } -- 5V --> low_level_controller
  high_level_controller -- Direction --> low_level_controller
  camera -- A/V --> high_level_controller
  high_level_controller -- UI --> screen
  low_level_controller -- Telemetry --> high_level_controller
  low_level_controller -- PWM --> motor_controller_left
  rail_14v_1@{ shape: junction } -- 14.8V --> motor_controller_left
  motor_controller_left --> motor_left
  motor_left --> sensor_left
  rail_5v_3@{ shape: junction } -- 5V --> sensor_left
  sensor_left -- SPI --> low_level_controller
  low_level_controller -- PWM --> motor_controller_right
  rail_14v_2@{ shape: junction } -- 14.8V --> motor_controller_right
  motor_controller_right --> motor_right
  motor_right --> sensor_right
  rail_5v_4@{ shape: junction } -- 5V --> sensor_right
  sensor_right -- SPI --> low_level_controller

Raspberry Pi 5 8GB / NVIDIA Jetson Orin Nano (8GB) for computer vision
STM32F3DISCOVERY with microcontroller, accelerometer, gyroscope, and motor output/input pins
4S Lipo for power
- Needs enough juice to run both of the control boards as well as the motors.
5V Step Down for main power rail
- I had originally planned for both a 5v and 3.3v rail, but decided to unify behind 5v for simplicity since the STM can safely interface with it.
Simple FOC Mini motor driver board
2804 BLDC + AS5600 Encoder kit for closed-loop motor control

Feedback

Before spending any money on this scheme, I wanted to get the input of folks much more knowledgeable than me. Besides, I was about to travel with my family to the UK and couldn’t immediately receive packages. I started sharing my plan on a few robotics Discord servers in search of critical feedback. The SimpleFOC folks gave me good advice:

@copper280z

I would change several things, don’t use the as5600 encoder, it’s not a good choice. Pick something that uses SPI or ABZ for an interface, like the mt6835, as5047, etc. ABZ is preferable for something like this because it’s got the lowest overhead.

Next, don’t use the stm32f3, use either an stm32f4, ideally one of the 168-180MHz variants like the f405 or f446, or use an stm32g4. The g4 is better but needs a bit more configuration for max performance because the stm32duino configuration for it is wrong/overly conservative.

The pi/jetson are overkill, but that’s fine. You really want the IMU to be connected to the stm32 and to have it run the balance control loop. The jetson is dramatically faster than the pi5, which might be nice if you want high res video or to run some ai type tasks.
@runger

I also think that for the size robot you’ll end up with, given battery, RPi/Jetson, etc you should go one size bigger with the motors, and use something like a 3506 or 4108 size motor

I really appreciate their feedback. @copper280z highlighted that the AS5600 uses either I²C or PWM output which consumes more CPU time than ABZ or SPI. ABZ has the lowest overhead, only changing one or two pins per incremental movement. They also suggested I choose a more powerful MCU (microcontroller). @runger said I needed a LOT more torque.

I narrowed down the list of STM boards based on this feedback and the prices at the time:

STM32F3DISCOVERY - $16.31 - STM32F303VC (72 MHz, 256KB) - but advised against
STM32F411EDISCOVERY - $15.55 - STM32F411VE (100 MHz, 512KB) - but a bit slower than recommended
STM32L562QE-DK - $75.77 - STM32L562QE (110 MHz, 512KB) - way too expensive and too many unwanted features
B-U585I-IOT02A - $64 - STM32U585AI (160 MHz, 2MB) - too expensive and too many unwanted features
STM32F407G-DISC1 - $20.48 - STM32F407VG (168 MHz, 1MB) - three axis only

I finally settled on the STM32F411DISCOVERY board based on my perception of the usefulness of the on-board hardware contrasted against the price. The AS5048A stood out as a good choice due to its ABZ and SPI modes. The larger of the motors, the GM4108H-120T, fit barely within the wheel dimensions of the Mk.II: perfect.


language: mermaid
graph TB
  battery[4S LIPO]
  splitter[XT90 Splitter]
  step_down[5V 5A step-down]
  rail_14v_in@{ shape: junction }
  rail_5v_in@{ shape: junction }
  battery -- 14.8V --> splitter
  splitter -- 14.8V --> rail_14v_in
  splitter -- 14.8V --> step_down
  step_down -- 5V --> rail_5v_in
  high_level_controller[Pi 5 / Jetson Orin]
  camera
  screen
  low_level_controller[STM32F411E-DISCOVERY]
  motor_controller_left[SimpleFOC Mini]
  motor_controller_right[SimpleFOC Mini]
  motor_left[GM4108 BLDC]
  motor_right[GM4108 BLDC]
  sensor_left[AS5047P]
  sensor_right[AS5047P]
  rail_5v_1@{ shape: junction } -- 5V --> high_level_controller
  rail_5v_2@{ shape: junction } -- 5V --> low_level_controller
  high_level_controller -- Direction --> low_level_controller
  camera -- A/V --> high_level_controller
  high_level_controller -- UI --> screen
  low_level_controller -- Telemetry --> high_level_controller
  low_level_controller -- PWM --> motor_controller_left
  rail_14v_1@{ shape: junction } -- 14.8V --> motor_controller_left
  motor_controller_left --> motor_left
  motor_left --> sensor_left
  rail_5v_3@{ shape: junction } -- 5V --> sensor_left
  sensor_left -- ABZ --> low_level_controller
  low_level_controller -- PWM --> motor_controller_right
  rail_14v_2@{ shape: junction } -- 14.8V --> motor_controller_right
  motor_controller_right --> motor_right
  motor_right --> sensor_right
  rail_5v_4@{ shape: junction } -- 5V --> sensor_right
  sensor_right -- ABZ --> low_level_controller

Coming Up…

This project is several firsts for me:

my first non-LEGO robotics project
my first CAD project
my first successful electronics project
my first 3D-printing project
my first soldering project

In the next post, I will cover the path to the first working prototypes.

Does this design ultimately change? (Spoiler: yes)

Next Post: Part 2: Prototypes

Running MVC5 Applications on Ubuntu Server (14.04.1 LTS)

Sun, 08 Feb 2015 00:00:00 +0000

We will be following the steps outline in the Install Mono on Linux guide and the ASP.NET 5 Getting Started guide.

When I installed Mono, KVM, and KPM as described in the above guides, I ran in to a couple of issues. First, unzip was not installed, causing the kvm upgrade command to fail. Secondly, libtool was not found, yielding the following exception:

System.DllNotFoundException: libdl
  at (wrapper managed-to-native) Microsoft.AspNet.Server.Kestrel.Networking.PlatformApis/LinuxApis:dlopen (string,int)
  at Microsoft.AspNet.Server.Kestrel.Networking.PlatformApis+LinuxApis.LoadLibrary (System.String dllToLoad) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.Load (System.String dllToLoad) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.KestrelEngine..ctor (ILibraryManager libraryManager) [0x00000] in <filename unknown>:0
  at Kestrel.ServerFactory.Start (IServerInformation serverInformation, System.Func`2 application) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.Hosting.HostingEngine.Start (Microsoft.AspNet.Hosting.HostingContext context) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.Hosting.Program.Main (System.String[] args) [0x00000] in <filename unknown>:0
  at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <filename unknown>:0

Installing the libtool package fixed the issue. Lastly, once libtool was installed, I was greeted with this exception:

System.NullReferenceException: Object reference not set to an instance of an object
  at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.loop_size () [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.Networking.UvLoopHandle.Init (Microsoft.AspNet.Server.Kestrel.Networking.Libuv uv) [0x00000] in <filename unknown>:0
  at Microsoft.AspNet.Server.Kestrel.KestrelThread.ThreadStart (System.Object parameter) [0x00000] in <filename unknown>:0

This was remedied by downloading, compiling, and installing libuv. In particular, libuv needs to be installed (or symlinked) to /usr/lib/libuv.so.1 for everything to work correctly.

So without further ado, here is a quick step-by-step guide to Running MVC5 applications on Ubuntu Server (14.04.1 LTS).

Install Mono

First, install Mono. These steps are pulled straight from the Install Mono on Linux guide.

language: sh
# Add the Xamarin public code-signing key
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF

# Add the Mono repository to apt sources
echo "deb http://download.mono-project.com/repo/debian wheezy main" | sudo tee /etc/apt/sources.list.d/mono-xamarin.list

# Install Mono
sudo apt-get update
sudo apt-get install -y mono-complete

Install libtool, libuv, unzip and Their Dependencies

Next we install the rest of the dependencies.

language: sh
# Install other prerequisites
sudo apt-get install -y autoconf build-essential git-core libtool unzip

# Compile libuv
cd /opt
sudo git clone https://github.com/libuv/libuv.git
cd libuv
sudo ./autogen.sh
sudo ./configure
sudo make
sudo make install
sudo ln -s /usr/local/lib/libuv.so /usr/lib/libuv.so.1

autoconf, build-essential, and git-core are necessary to build and install libuv.

Install the K Version Manager and the K Runtime Environment

These steps are pulled from the ASP.NET 5 Getting Started guide.

language: sh
# Get the K Version Manager
curl -sSL https://raw.githubusercontent.com/aspnet/Home/master/kvminstall.sh | sh && source ~/.kre/kvm/kvm.sh

# Install the runtime
kvm upgrage

All Done?

Here we could call it a day, since your Ubuntu system is now set up to run ASP.NET 5 applications. Why don’t we follow through with the rest of the guide so we can see it working?

language: sh
# Obtain the ASP.NET 5 samples
git clone https://github.com/aspnet/Home.git ~/AspNetHome
cd ~/AspNetHome/samples/HelloMvc

# Get all of the .NET dependencies necessary to run the sample
kpm restore

# Run the sample
k kestrel

Here it is, running on my machine:

Range header, I choose you (for pagination)!

Wed, 21 Nov 2012 00:00:00 +0000

I was looking around for a semantic way to do pagination over HTTP recently. I had previously used the OData-style $skip and $top query string parameters and I was just generally dissatisfied with how that turned out. So, I was searching for alternatives.

I happened upon a question on Stack Overflow that discussed using the Range HTTP header for pagination purposes. The conclusion there was to use a different approach (mimicking Atom), but I disagree. I feel that the Range header is a perfect fit for pagination.

Why is the Range header a good fit?

The Range header is normally used by browsers to request specific byte-ranges of binary files. It powers the pause-and-resume functionality of almost every download manager in existence. This fact tempted me to simply conclude that usage of the Range header for any purpose other than pause-and-resume would be an abuse.

However, RFC 2616 has this to say about the Range header:

HTTP retrieval requests using conditional or unconditional GET methods MAY request one or more sub-ranges of the entity, instead of the entire entity, using the Range request header, which applies to the entity returned as the result of the request.

Our entity is a collection of objects, and we are requesting “sub-ranges of the entity, instead of the entire entity.” This is a perfect fit.

How does it measure up?

So, from the spec, it looks like the Range header was designed to handle this concern, but how does this play out in practice? It would be foolish to just commit to using the header without making sure it was up to the task.

Let’s evaluate its behavior in contrast to OData-style pagination.

One thing to note:
These examples contain partial HTTP requests. Common elements between the OData and Ranger header styles have been factored out. These include the http version; the Host, Accept, Content-Type, and Content-Length headers; and the data itself.

Querying the root of the collection.

OData-style

language: http
GET /users

language: http
200 OK

[ 0, …, 9 ]

Range-header-style

language: http
GET /users

language: http
200 OK
Accept-Ranges: users
Content-Range: users 0-9/200

[ 0, …, 9 ]

Comparison

Both of these have defaults for what the user is allowed to request (limited to 10), but the Range header style is allowed to send back the count, without $inlinecount being specified. The Accept-Ranges header signals that the users range is accepted and implies that a Content-Ranges may be present. This makes for good discoverability.

Importantly, the response code for the Range header style is 200 (rather than 206 as you might expect) because the request did not include the Range header itself. Accept-Ranges and Content-Range headers are still allowed in the response, keeping it semantically correct.

The OData-style, in contrast, will not automatically include the count of the collection in the response. Even if this response were to include the count, unsolicited, there is still no easy way to discover how this data is sent.

Pulling the first page of results normally.

OData-style

language: http
GET /users?$skip=0&$top=10&$inlinecount=allpages

language: http
200 OK
X-Some-Side-Channel: count=200

[ 0, …, 9 ]

Range-header-style

language: http
GET /users
Range: users=0-9

language: http
206 Partial Content
Accept-Ranges: users
Content-Range: users 0-9/200

[ 0, …, 9 ]

Comparison

Since the request included the Range header, the server is allowed to respond with a 206 Partial Content status code, indicating the presence of the Content-Range response header.

OData had to specify the $inlinecount=allpages parameter in order to get the full length of the filtered collection, bloating the URL.

Additionally, the Range header style is allowed to return a Content-Range response looking something like this:

language: http
Content-Range: 200-250/*

This indicates that the full count is not included, possibly because it is too expensive to calculate. This is often the case for complex queries. This flexibility allows the server to chose whether or not to calculate the total length, based on the particular query at hand. In the OData version, however, the $inlinecount parameter is a command to get the count. The server is not free to withold the count when it is expensive or difficult to obtain.

Pulling subsequent pages of data.

Both systems perform roughly the same for pages in the middle of the collection. Neither system has anything to offer here, nor does either fail in any special way.

Requesting past the end of the collection.

OData-style

language: http
GET /users?$skip=1000

language: http
200 OK

[]

Range-header-style

language: http
GET /users
Range: users=1000-

language: http
416 Requested Range Not Satisfiable

Notice that the unbounded range mimics the semantics of the OData-style request.

Comparison

The response of the Range header style request is precisely in line with the spec:

A server SHOULD return a response with this status code if a request included a Range request-header field, and none of the range-specifier values in this field overlap the current extent of the selected resource, and the request did not include an If-Range request-header field. (For byte-ranges, this means that the first-byte-pos of all of the byte-range-spec values were greater than the current length of the selected resource.)

Note, as well, that byte-ranges are specified here as a single use-case of the range header, firmly implying that the usage as shown here is the correct semantic usage.

The OData-style is not allowed to return a 416 status code, because it never sent a Range header. One could argue, however, that this status code is semantically correct, even for the OData style request.

Additional functionality of the range header.

Here are some things that the Range header supports that have no analog in the OData-style.

Discoverability

language: http
OPTIONS /users

language: http
200 OK
Accept-Ranges: users

The Range header style allows for discovery of acceptable ranges via the OPTIONS HTTP verb.

Last-n requests

language: http
GET /users
Range: users=-5

language: http
206 Partial Content
Accept-Ranges: users
Content-Range: users 195-199/200

[ 195, …, 199 ]

The Range header style allows for requests rooted at the end of the entity, rather than the begining.

Multipart ranges

language: http
GET /users
Range: users=0-9,50-59

language: http
206 Partial Content
Accept-Ranges: users
Content-Type: multipart/mixed; boundary=next

--next
Content-Range: users 0-9/200

[ 0, …, 9 ]

--next
Content-Range: users 50-59/200

[ 50, …, 59 ]

--next--

The Range header style allows for requests that specify multiple ranges; HTTP supports sending results back as a multipart document.

The If-Range header

The If-Range header allows a range of the entity to be downloaded if it has not changed, but download the entire entity if it has changed. These semantics don’t really fit the pagination model very well, since it is pretty much aimed at resuming paused downloads.

Infinite or indeterminate collections

language: http
GET /users?name=Fred

language: http
206 Partial Content
Accept-Ranges: users
Content-Range: users 0-100/*

[ 0, …, 100 ]

As mentioned earlier, the Content-Range response header can specify * as the total size of the entity. This means that the entity has an unknown or unbounded size. This is particularly useful in search applications that may not know the full size of the result set for every query.

Conclusion

To me, it seems that the Range header is both more powerful and more semantically correct than the OData-style of query string parameters (or any query string style, for that matter). As such, I will always use the standard Range HTTP header whenever I implement REST APIs.

I urge you to do the same.

Getting a Proper MVC Date-picker using jQuery UI

Sat, 03 Sep 2011 00:00:00 +0000

There are plenty of tutorials out there on how to get date-pickers to work in MVC, but most of them that I have seen fall short for some reason or another. Here are my requirements for a “proper” date-picker in MVC:

It must be automatic, when using the EditorFor family of methods.
It must degrade gracefully. That is, it should fall back to a regular text box for users without JavaScript.
It must always work for any complex model.
It must round-trip the data entered, even if the data is an invalid date.
It must be self-contained. That is, it should not require a script at the top of the page to work.

Some demos fail to work for complex models, and most require a surrogate script at the top of the page. We’ll see how to avoid those problems in the coming sections. I will be assuming that your project already has jQuery and jQuery UI.

Automatic Editor

The ability for MVC to automatically pick an editor based on the data-type has been covered at length before, just search Google for “EditorTemplates” if you are looking for examples.

Quickly, then, here is how to add an automatic template for any type:

Navigate to the ~/Views/Shared folder of your web project.
Create a folder named “EditorTemplates”, if it does not already exist.
Inside of this folder, create a partial view with the name of your type. (DateTime, in our case).

We will also choose to have a strongly-typed view, using DateTime? as our type. This will allow it to work for both nullable and non-nullable dates

Here is how the folder structure should be laid out:

Date Picker - 1 - Layout

Now, MVC will automatically select our editor whenever it has a DateTime or a Nullable<DateTime> to render.

Round-trippin’ With My Two Favorite Allies

We need to use a regular HTML text box for our date picker for two reasons:

It will allow for date-input, even if the user has disabled JavaScript.
MVC will automatically round-trip the data and perform model binding, if it is rendered in a certain way.

We chose to use a dynamic partial view, so that we could round-trip textual data, above, so the simplest way to get the editor working is this, placed in the DateTime.cshtml file:

language: cshtml
@model DateTime?
@Html.TextBox("", Model)

Interestingly, we are passing an empty string in for the “name” parameter for the text box. This is the secret sauce for getting MVC to render the proper name to support complex models and for it to automatically round-trip the model for. Second, we are simply passing the model as the value of the text box. MVC will take this and call ToString() on the value, which will give us the default formatting for dates.

We may, however, want dates to be shown without their time component. To do that, we need to override the default formatting to our liking:

language: cshtml
@model DateTime?
@Html.TextBox("", Model.HasValue ? Model.Value.ToString("d MMM yyyy") : "")

This will not prevent the users’ input from round-tripping, but will instead provide a default formatting for views where there is no posted data to reference, e.g. loading the edit view for the first time.

Self-contained Script

Some people prefer to have a single script for wiring-up their date-pickers at the top of the page. I find this to be less than ideal, because I usually don’t date pickers on a page, so it doesn’t belong in the master layout. I also don’t want to duplicate the code on every page, because it means that a simple change to the date picker would have to be performed in many places, and it could easily be forgotten on a page, preventing date pickers from working there.

With that in mind, my preference is to have a self-contained script for creating the date-picker immediately following the text box. The challenge that we face when we want to do this is coming up with a way to refer to the specific text-box in a jQuery selector. The easiest way, from a jQuery standpoint, would be to use the ID of the text box.

Luckily, MVC allows us access to the field name using the Html.ViewContext.ViewData.TemplateInfo.GetFullHtmlFieldName method. This is, in fact, the way that MVC assigns the name and ID to the text box. However, there is no built-in way to turn that name into an ID. A quick perusal of the MVC source code shows that the ID is a simple text replacement. We can precisely emulate their text replacement with a regular expression replacement. Here is the completed control:

language: cshtml
@model DateTime?
@{
    var name = Html.ViewContext.ViewData.TemplateInfo.GetFullHtmlFieldName(string.Empty);
    var id = System.Text.RegularExpressions.Regex.Replace(name, @"[^-\w\d:_]", HtmlHelper.IdAttributeDotReplacement);
}
@Html.TextBox("", Model.HasValue ? Model.Value.ToString("d MMM yyyy") : "")
<script type="text/javascript">$(function () { $("#@id").datepicker({ dateFormat: 'd M yy' }); });</script>

Now, we have a feature-complete, self-contained date picker, that is automatically applied for all DateTime fields!

Getting JavaScript IntelliSense in MVC Razor Views

Thu, 01 Sep 2011 00:00:00 +0000

A while ago, I posted about Getting IntelliSense working in ASP.NET MVC 3. Since the release MVC 3, the primary view engine has changed to Razor, and the previous technique isn’t as elegant.

With that in mind, I searched for a slightly prettier way to get IntelliSense working in Razor views. I was, unfortunately, unable to get the scripts to be referenced in the views automatically, but I think that this way is clean enough.

I used a @section directive to scoop out a part of the page during rendering, in order to prevent the “vsdoc” scripts files from being rendered to the final output. Here is how it looks in action:

But now we have a couple of challenges to deal with. If we just ignore the section, we get this:

The following sections have been defined but have not been rendered for the layout page “~/Views/Shared/_Layout.cshtml”: “intellisense”.

And we know that we don’t want to render it to the final output, so we have to come up with a way to trick the Razor engine into thinking that we did, in fact, render the output.

My first attempt at solving this was to put this into the _Layout.cshtml page:

language: cshtml
@{RenderSection("intellisense", required: false);}

But that was not enough to trick the view engine. Apparently, you have to actually render the section somewhere in order for it to be considered “rendered. Luckily, we can just render it to a throw-away StringWriter, like so:

language: cshtml
@{WriteTo(new StringWriter(), RenderSection("intellisense", required: false));}

That is just enough to get it to work properly and display the IntelliSense as desired.

I am still looking for a way to get this to happen automatically, but for now, placing the @section intellisense region at the top of each view will have to do.

Slick Analytics with LogParser and SqlBulkCopy

Wed, 31 Aug 2011 00:00:00 +0000

Slick Analytics

What do you do when you need to add analytics to your server? Well, Google Analytics is a good option, for sure, but it doesn’t capture important, juicy details like bandwidth usage, time to complete requests, cache-hits and so on. If you need data like this, the best source is going to be the web access logs from your HTTP server.

I’ve taken a look at a few 3rd party tools for creating analytics from these log files, and there are downsides to every one of them. Cacti is a pretty good option, however it requires a few things that may not be installed on a Windows server (i.e. PHP, MySQL, RRDTool, and net-snmp). So, based on the fact that most existing solutions aren’t designed to fit seamlessly into a windows environment, I decided to come up with my own solution.

My solution is based on a couple of off-the-shelf Microsoft products that, if you are running on the Microsoft stack, should already be licensed to you. Here’s the tool chain:

IIS
Log Parser 2.2 (For turning W3C logs into an easy-to-import format. It may also support Apache W3C logs, but I haven’t tested it.)
some secret sauce
SQL Server
SQL Server Reporting Services (or your favorite reporting suite)

So, what is the secret sauce? Well, SQL Server’s ADO.NET provider exposes the SQL Server Bulk Import API through a class called SqlBulkCopy. We can easily craft a C# program to take the clean, consistent output of Log Parser and stream it into SQL Server at breakneck speeds.

Log Parsing

First, let’s get Log Parser humming. Microsoft’s Log Parser accepts SQL-like commands from the command line, and, depending on the particular command, can output to a variety of text formats, execute SQL statements, or create images. Here is an example of the type of SQL Statement we are looking for:

language: sql
SELECT * INTO log1.csv FROM C:\WINDOWS\system32\LogFiles\W3SVC1\ex110101.log

However, as you may be able to tell from the file path, we need to parameterize the source for different IIS sites (e.g. W3SVC9999) and dates (e.g. ex110228.log). Here is an example command script that can take care of those variables:

language: batch
@echo off
set logparser="C:\Program Files\Log Parser 2.2\LogParser.exe"

set siteid=%1
if "%siteid%"=="" set siteid=1
set logdate=%date:~-2%%date:~4,2%%date:~7,2%
%logparser% "SELECT * INTO log-w3svc%siteid%-ex%logdate%.csv FROM C:\WINDOWS\system32\LogFiles\W3SVC%siteid%\ex%logdate%.log"

If you run that, you should end up with a CSV file in the current directory, with the current day’s logs for the default IIS website. It also accepts the website ID number as a parameter on the command line, if you want to run this for more than one site.

Now, here I would like to note that Log Parser supports a mode of operation in which it remembers where it left off in a log file, and skips there on subsequent runs. However, I have found that, since IIS does not flush its log files and since Windows caches writes, the log will sometimes end half-way through an entry. When a situation like this arises, Log Parser gets confused and completely fails to properly parse any further entries. In addition, its SQL support is useful but is unable to do the incremental loads that we would like to do.

Set-up the Database

We need a place to shove that data, but we have two masters to server at this point. On one hand, we want to transfer as little data to the database server as possible, for obvious reasons. On the other, we don’t want to have long-running transactions against the main IIS logs table, since it is really made for OLAP and response time of queries is paramount. This all leans toward a scheme of staging and merging data, rather than cherry-picking new rows to import. And, even though we don’t want to transfer tons of data, we should be OK with 1-day’s-worth in a staging table. After the data is staged, it can be incrementally moved into the main storage table to reduce the impact on the analytics.

So, we will need two (very similar) tables during our import process: the staging table, and the main storage table. The main storage table should match the W3C Log format, like so:

language: sql
CREATE TABLE [dbo].[w3clog]
(
    [RowId] bigint IDENTITY(1,1) NOT NULL PRIMARY KEY,
    [LogFilename] varchar(255) NOT NULL,
    [LogRow] int NOT NULL,
    [date] date NULL,
    [time] time(0) NULL,
    [datetime] AS (CONVERT(datetime2(0), date + CONVERT(datetime, time, 0), 0)) PERSISTED,
    [c-ip] varchar(50) NULL,
    [cs-username] varchar(255) NULL,
    [s-sitename] varchar(255) NULL,
    [s-computername] varchar(255) NOT NULL,
    [s-ip] varchar(50) NULL,
    [s-port] varchar(255) NULL,
    [cs-method] varchar(255) NULL,
    [cs-uri-stem] varchar(2048) NULL,
    [cs-uri-query] varchar(max) NULL,
    [sc-status] int NULL,
    [sc-substatus] int NULL,
    [sc-win32-status] bigint NULL,
    [sc-bytes] int NULL,
    [cs-bytes] int NULL,
    [time-taken] bigint NULL,
    [cs-version] int NULL,
    [cs-host] varchar(255) NULL,
    [cs(User-Agent)] varchar(1000) NULL,
    [cs(Cookie)] varchar(max) NULL,
    [cs(Referer)] varchar(2000) NULL,
    [s-event] varchar(255) NULL,
    [s-process-type] varchar(255) NULL,
    [s-user-time] int NULL,
    [s-kernel-time] int NULL,
    [s-page-faults] int NULL,
    [s-total-procs] int NULL,
    [s-active-procs] int NULL,
    [s-stopped-procs] int NULL
)

The staging table will be an almost identical temp table, with the single omission of the computed ‘datetime’ column.

Build the Importer

So, now it is time to explore the power of the SqlBulkImporter class.

This code is available on GitHub, so there is no need to follow along unless you are building a custom version. In light of this, I will be skimming over a few of the easier parts.

Create the Connections

Our importer needs to be able to read the CSV files, and spit the data into SQL server. Luckily, both of these are supported by Microsoft’s ADO.NET providers. First, we will use a standard SQL Server connection for the destination database:</p>

Next, we will use an ODBC connection for the CSV files:

language: text
Driver={Microsoft Text Driver (*.txt; *.csv)};Dbq=.;Extensions=csv

Using the Microsoft Text Driver, it is possible to read CSV files like this:

language: csharp
var table = new DataTable();
using (var connection = new OdbcConnection(Settings.Default.ImportDriver))
{
    var dataAdapter = new OdbcDataAdapter("select * from log1.csv", connection);
    dataAdapter.Fill(table);
}

Streaming the Data

The little example above is fine-and-dandy for reading the data into memory, but just for the sake of future-proofing, we should aim to stream the data in. This will ensure that we can import successfully, even during extreme loads. Rather than using a DataAdapter and DataTable, we will opt to read the rows and columns ourselves with the ExecuteReader family of methods.

To clear up any uncertainty, here is the order and nesting of the various operations that will need to take place for a successful, performant import:

Connect to the destination database
- Begin a transaction
  - Create a temporary “staging” table for the logs
  - For each CSV file of interest:
    - Open a reader for the CSV file
      - Bulk-import the data from the CSV file into the staging table
    - Close the CSV file
  - Merge the contents of the staging table into the main storage table
- Commit the transaction
Disconnect from the destination database

There are only two noteworthy bits: the bulk import itself and the merge operation.

The Bulk Import

The code I use to do the import is fairly simple:

language: csharp
private static void BulkImport(IDataReader reader, SqlTransaction transaction)
{
    using (var importer = new SqlBulkCopy(transaction.Connection, SqlBulkCopyOptions.Default, transaction))
    {
        for (int field = 0; field < reader.FieldCount; field++)
        {
            var name = reader.GetName(field);
            importer.ColumnMappings.Add(name, name);
        }

        importer.DestinationTableName = "#w3clog_staging";
        importer.WriteToServer(reader);
    }
}

Simple as it is, there is a lot that we can learn from this compact section of code:

The SqlBulkCopy class can be given options to create an internal transaction, fire (or bypass) triggers, fire (or bypass) check constraints, insert into identity columns, and etc.
Bulk SQL operations can enter into external transaction, and can be rolled back in the same way that any transaction can. I was expecting this, but was still pleasantly surprised to find that SQL Server inherently supports this.
SQL Bulk Copy operations use an ordinal column mapping by default. This means that columns are mapped based on their position in the import data and in the table rather than their names. This can be overridden by mapping each column pair by some combination of ordinal position and column name. Here, we are using a simple name-to-name mapping.
Bulk operations can be performed against temp tables. I was not expecting this to work, and was again pleasantly surprised.
The SqlBulkCopy class can use either a set of DataRows (from a DataTable, for example) or an IDataReader as its source.
ADO.NET does not have a time data type, it uses DateTime instead. In addition, the bulk importer does not expose any way to manually convert the columns. Therefore, either the destination table must use datetime for all time columns, or the source reader must expose the data as varchar. (It would also be possible to wrap one IDataReader with another that did the mapping, but this is more work than it is worth, in practice.)

The Merge Operation

The final notable part of the program is the merge operation. There are several ways to get this done; from a single, complex insert statement, to individual insert operations.

We will be aiming for a set-based approach, utilizing joins to do most of the heavy lifting.

First, to delete entries that already exist in the main table, we issue a command like this:

language: sql
DELETE
    s
FROM
    [#w3clog_staging] s
INNER JOIN
    dbo.[w3clog] p
  ON
    s.[LogFilename] = p.[LogFilename]
  AND
    s.[LogRow] = p.[LogRow]

Simple stuff really, but to help it out, we will probably want a unique index on the main table: (this only needs to be done once)

language: sql
CREATE UNIQUE NONCLUSTERED INDEX [UK_w3clog_LogRow] ON dbo.[w3clog] 
(
    [LogFilename] ASC,
    [LogRow] ASC
)

Next we need to delete duplicate entries that may have crept into the staging table:

language: sql
DELETE FROM
    [#w3clog_staging]
WHERE
    [RowId] IN
      (
        SELECT
            RowId
        FROM
          (
            SELECT
                RowId,
                ROW_NUMBER() OVER (PARTITION BY [LogFilename], [LogRow] ORDER BY [RowId]) [Instance]
            FROM
                [#w3clog_staging]
          ) instances
        WHERE
            [Instance] > 1
      )

This uses the SQL Server Window Function ROW_NUMBER() to determine individual rows to delete.

The final action is to move the data into the main table:

language: sql
INSERT INTO
    [w3clog]
    (
        [LogFilename], [LogRow], [date], …
    )
SELECT
    [LogFilename], [LogRow], [date], …
FROM
    [#w3clog_staging]

Done! Now our whole import process is complete. Let’s see if we can turn this into pretty graphs…

Analyzing the Data

Let’s get a simple graph showing the last 3 day’s hits, upload, and download, grouped into hourly buckets.

First, the query:

language: sql
SELECT
    DATEPART(hour, [time]),
    COUNT(*) [hits],
    SUM([sc-bytes]) [upload-bytes],
    SUM([cs-bytes]) [download-bytes]
FROM
    dbo.[w3clog]
WHERE
    [datetime] >= DATEADD(day, -3, GETUTCDATE())
GROUP BY
    DATEPART(hour, [time])

Simple stuff, right?

So, here is how my server’s graph looks: (in Excel, since I’m using SQL Express on my server)

Can we find out what bots are hitting the site? Sure:

language: sql
SELECT
    [cs(User-Agent)],
    COUNT(*) [hits],
    COUNT(DISTINCT [c-ip]) [ips]
FROM
    [w3clog]
WHERE
    [cs(User-Agent)] LIKE '%http%'
GROUP BY
    [cs(User-Agent)]
ORDER BY
    [hits] DESC,
    [ips] DESC

Here are my results:

User Agent	Hits	Distinct IPs
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)	71624	112
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)	60536	441
Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)	32636	329
Mozilla/5.0 (compatible; Baiduspider/2.0; http://www.baidu.com/search/spider.html)	23749	225
Mozilla/5.0 (compatible; YandexBot/3.0; http://yandex.com/bots)	16696	7
webnumbrFetcher/1.0 (http://webnumbr.com/)	16551	1
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)	14147	5
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)	13892	61
Baiduspider ( http://www.baidu.com/search/spider.htm)	10445	550
Sosospider ( http://help.soso.com/webspider.htm)	8280	76
Mozilla/5.0 (compatible; SiteBot/0.1; http://www.sitebot.org/robot/)	7614	5
Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php? )	7085	241

Interesting!

I have included many more queries to play with in the project on GitHub. Stay tuned for more!

CoffeeDemo – A Simple Demo of IronJS, using CoffeeScript

Wed, 29 Jun 2011 00:00:00 +0000

Warning: the APIs used in this demo are subject to improvement in the future. This is just a demo to get your feet wet with the 0.2.0.1 release of IronJS. IronJS is still in the early stages, so your mileage may vary.

How would you like to have a CoffeeScript compiler, running 100% managed code? Well, I’ll leave out the foreplay and get right to it.

Fire up an instance of Visual Studio, with NuGet installed.

This is pretty self-explanatory, but just in case you don’t have NuGet installed, you can follow the directions on the NuGet website to get started.

Create a new Windows Application.

Create a new Windows Application and name it ‘CoffeeDemo’.

Use NuGet to install IronJS.

For those of you experienced with NuGet, the command in the Package Manager Console is:

language: power-shell
Install-Package IronJS

If you prefer to use the GUI to add your package, you can consult the GUI directions provided at NuGet.org

Lay-out your project and form.

At a minimum, you should add the following to your form:

An input textbox, for CoffeeScript.
An output textbox, for JavaScript.
A button, for executing the JavaScript.

You will want to make your text boxes Multiline, and large enough to enter some JavaScript source.

Here is how mine looks:

I decided to use a split panel (anchored to all four sides) to contain my text boxes. I set my text boxes to dock-fill and gave them both horizontal and vertical scroll bars. I also anchored my Execute button to the bottom right. Finally, I hate the name Form1 , so I always rename my form to View or MainView.

Name your controls CoffeeScriptBox, JavaScriptBox, and ExecuteButton.

Create the IronJS context.

In the code for our view, we need to create an IronJS context, to do all of the heavy lifting behind the scenes. We will do this with a function InitializeContext, called from the constructor:

language: csharp
using System;
using System.Windows.Forms;
using IronJS;
using IronJS.Hosting;
using IronJS.Native;

namespace CoffeeDemo
{
    public partial class View : Form
    {
        private CSharp.Context context;

        public View()
        {
            InitializeContext();
            InitializeComponent();
        }

        private void InitializeContext()
        {
            Action<string> alert = message => MessageBox.Show(message);
            this.context = new CSharp.Context();
            this.context.SetGlobal("alert", Utils.CreateFunction(this.context.Environment, 1, alert));
        }
    }
}

Here, we can see the basics of creating a contexts, as well as exposing a CLR function to JavaScript. The alert function is now wired up to call directly into MessageBox.Show.

Wire-up the execute button.

Just wire up the Click even of the execute button to a function like this,

language: csharp
private void ExecuteButton_Click(object sender, EventArgs e)
{
    try
    {
        this.context.Execute(this.JavaScriptBox.Text);
    }
    catch (IronJS.Error.Error ex)
    {
        MessageBox.Show(ex.ToString(), "Error from IronJS");
    }
}

You can run a test like this:

Automatically download CoffeeScript.

There are a couple of different methods that we could use to bring CoffeeScript into our app. We could include it in the folder, we could add it as an assembly resource, etc.

My preference for this demo, however, is to download CoffeeScript to the /bin directory on the first run of the app.

language: csharp
…
using System.IO;
using System.Net;
…
namespace CoffeeDemo
{
    public partial class View : Form
    {
        …
        public View()
        {
            …
            LoadCoffeeScript();
        }
        …
        private void LoadCoffeeScript()
        {
            if (!File.Exists("coffee-script.js"))
            {
                var client = new WebClient();
                client.DownloadFile("https://raw.github.com/jashkenas/coffee-script/master/extras/coffee-script.js", "coffee-script.js");
            }

            this.context.ExecuteFile("coffee-script.js");
        }
    }
}

At this point CoffeeScript is ready to go, and can be tested in the JavaScript text box. We are loading it in the constructor, which could negatively impact the application start-up time. However, for this example, I’m not concentrating on performance.

Try this for an example:

language: javascript
alert(CoffeeScript.compile('a = (x) -> x * x', { bare: true }));

Wire-up the TextChanged event for live updating.

Add a field of type FunctionObject to the view’s class, and add this function as the TextChanged event on the CoffeeScript text box:

language: csharp
private FunctionObject compile;
…
private void CoffeeScriptBox_TextChanged(object sender, EventArgs e)
{
    if (this.compile == null)
    {
        this .context.Execute("var compile = function (src) { return CoffeeScript.compile(src, { bare: true }); };");
        this .compile = this.context.GetGlobalAs<FunctionObject>("compile");
    }
     
    try
    {
        var boxedResult = this.compile.Call(this.context.Globals, this.CoffeeScriptBox.Text);
        var result = TypeConverter.ToString(boxedResult);
        this.JavaScriptBox.Text = result.Replace("\n", "\r\n");
    }
    catch (IronJS.Error.Error)
    {
    }
}

This shows you how to call a JavaScript function from the CLR.

CoffeeDemo in action!

You should now have a working, live-updating CoffeeScript compiler. You will almost certainly notice the compilation lag on the first key-press where we are compiling the helper function, and calling the CoffeeScript compiler for the first time. The lag is due to IronJS pushing everything into an in-memory, dynamic assembly. Subsequent key-presses should be pretty quick, tho, and we are striving to make it faster.

Stay tuned.

We are actively working on better .NET integration so that you can use native .NET types, all without using wrapper functions. So, stay with us, and please follow us on Github!

Getting IntelliSense in JavaScript files in ASP.NET MVC

Wed, 29 Dec 2010 00:00:00 +0000

The Problem

If you are like me, you have probably had trouble in the past trying to get JavaScript IntelliSense to work well in Visual Studio.

You have probably gone through these steps:

“I’ll just add my script here, and presto!”

“Well, that didn’t work. It kept the App Path Modifier… dammit.”

“I’ll just add a run-at attribute, so that ASP.NET transforms the path for me…”

“Oh, my goodness gracious!”

So where does that leave us?

We can’t use runat="server" on script tags, because ASP.NET has repurposed this format in order to include server-side code into the page. This in effect means that we can’t use the “virtual path” to reference the same script from everywhere in our app. Since we can’t use a virtual path, and we can’t use a relative path, we are stuck.

Add on top of this that we often need to switch between different scripts for auto-completion, debugging, and production and you start to see the direness of the situation.

At the center of the issue is the fact that Visual Studio can’t execute code in the editor in order to “know” where a script will come from. If it could, then we would just write a chunk of code to do the virtual path transformation based on current mode. We would have minified scripts for production (by default), VSDoc scripts for IntelliSense (in design mode), and full source scripts for debugging (when DEBUG is defined or a debugger is attached).

At this point, I used to conclude that I had to live with all of my .aspx files living in one folder, or I had to live without JavaScript IntelliSense.

A Solution

The little bit of magic required to make everything work in our favor is the fact that Visual Studio will treat everything outside of <% ... %> code sections as though it were one continuous block of HTML. This means that we can use a technique similar to conditional comments in order to ignore sections of HTML, but still have it recognized as part of the HTML document.

Take these lines for example:

language: html
<% if (false) { %>
    Test
<% } %>

After being transformed by ASP.NET into a pure C# listing, this code will be compiled into an assembly. Since the code is unreachable, it will be ignored and will not be checked for definite assignment as per the C# spec (§5.3.3.1).

Extending this idea a little bit further, we can add our long-sought-after IntelliSense:

language: html
<% if (false) { %>
    <script src="../../Scripts/jquery-1.4.1-vsdoc.js" type="text/javascript"></script>
<% } %>

We can use the relative path here, since this will never make it across the wire. Just make sure that the script is referenced relative to the physical file in which this tag exists.

Now, for the real script tag, we can use the static functions that ASP.NET provides to emit the proper path:

language: html
<script src="<%: Url.Content("~/Scripts/jquery-1.4.1-vsdoc.js") %>" type="text/javascript"></script>

For ASP.NET WebForms, you would use the Server.ApplyAppPathModifier() function in place of Url.Content().

Variations and Conclusion

Variations on this trick would include using compiler directives like #if FALSE, or /* block comments */ to exclude the section of code. Both of these methods seem to work.

If you use the method described above with a build server, or while compiling view locally, you may get a compiler warning saying “unreachable code detected.” If this warning bothers you you can use #pragma warning disable 162 and #pragma warning enable 162 at the start and end of your if statement to suppress it.

I hope this helps you get the most out of Visual Studio and ASP.NET! Your feedback is welcomed.

Pain Free T4MVC

Thu, 23 Dec 2010 00:00:00 +0000

Here is how to get T4MVC to run every time you build, without the mark-as-dirty hack or any Visual Studio support:

Download and install the prerequisites.

Unfortunately, Plain-old-VS2010 does not have the ability to run T4 Templates on every build built-in, so we have to rely on an external toolkit to help us along. On the bright side, it’s just the Visual Studio SDK (which you might already have) plus an additional little tidbit.

These must be installed in order, and you should probably exit VS to do so:
1. Visual Studio SDK
2. Visualization and Modeling SDK
Start Visual Studio and open your project file for editing.

Right-click on the project in question, and choose “Unload Project”; this will make it show up as “(unavailable)”. Now, you can right-click it again and choose “Edit YourProject.csproj”.

Add the relevant sections to your project file.

Set the following properties in your project:

language: xml
<MvcBuildViews>true</MvcBuildViews>
<TransformOnBuild>true</TransformOnBuild>

The result should look something like this:

Add the toolkit magic as an MSBuild import:

language: xml
<Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\TextTemplating\v10.0\Microsoft.TextTemplating.targets" />

Again, here is a visual of the result:

Save your changes and reload your project.

Right-click your project, which should still be showing as “(unavailable)”, and Voila! Your T4 templates will re-gen on every build!

“That’s pretty great,” you say, “but will it work on my build server?”

Yes, it will work! You simply need to follow the MSDN instructions for this. Just a few files need to be copied on to the build server.

I hope this helps people get the most out of MVC. Your feedback is welcomed.