Tech Talk

UTF-8 in Python

2017-11-07T03:16:00.001-08:00

I have been working a lot with python lately. Especially building tools that convert from Excel to JSON and XML. The data have been translations and similar, so encoding has been a real issue. Generally, it works well in python, but when working in a Windows environment, it is easy to go wrong among the different encoding standards such as "UTF-8" and CP1252 and so on. I won't pretend to be an expert on this at all. I have just tried to get the stuff working without having to understand too much of it. So if you have any useful background information about the snippets below, please go ahead and share a comment.

Today this stole a couple of hours from me again, so I write this down as a memory annotation... once and for all!

When using openpyxl, all strings are converted to "UTF-8" when reading the data from the workbook, so there is no need to do any conversion there or try to find any obscure encoding settings in Excel. This far all strings, Chinese, Swedish or Hungarian, have arrived correct.

To Terminal

Getting it correct when writing them to the terminal was a bit of a trick though. The trick seems to be to always write the text as bytes. To do this to stdout I had to this:

sys.stdout.buffer.write(message.encode('utf-8'))

To JSON

When generating JSON-files it is important to make sure the json generator generates the right content by setting ensure_ascii=False.

Then write the file as a binary file. Whatever I do, things seem to go wrong if I decode the string before writing it.

with open(fileName, mode="wb") as jsonFile:
jsonFile.write(bytes(json.dumps(dict_to_write, ensure_ascii=False), 'utf-8'))

To XML

Again, create a binary file and write an encoded byte array rather than decoding to a string. I have used pybx to create bindings from the XSD files. Pybx builds on the minidom libraries... I think.

from pyxb.namespace.builtin import XMLSchema_instance as xsi

bds = pyxb.utils.domutils.BindingDOMSupport()
bds.declareNamespace(xsi)
dom = self._root.toDOM(bds)
bds.addAttribute(dom.documentElement, xsi.createExpandedName('schemaLocation'), "http://my_target_ns/ns1 ../xsd/my_schema.xsd")
bds.addXMLNSDeclaration(dom.documentElement, xsi)

if self._root is None:
raise Exception("There is no root element to write to file.")

if self._fileName is None:
self._fileName = os.path.join(os.getcwd(), 'xml/' + self._root.name + '.xml')

with open(self._fileName, 'wb') as xmlfile:
xmlfile.write(dom.toprettyxml(indent=' ', encoding='utf-8'))

Good Luck!

Clock setup for Olimex E407 using STM32CubeMX

2015-08-10T23:11:00.002-07:00

Recently I got a reason to dig up my old Olimex E407 board and the RS232 shield I made for it (see earlier posts on this blog). This time though, I'm using Keil in Windows together with a ST Link V2. Getting the clocks and pll's right isn't an easy task, and certainly not a fun one, so I decided to give CubeMX and ST's HAL libraries a try.

The STM32CubeMX program (download it from ST.com) is a fairly new invention, and the libraries leave a lot to whish for still, so you should be careful with it. The libraries are still ful of bugs, but they are also getting a lot better rapidly.

Ok, first of course, download and install STM32CubeMX. If you want to launch it from Keil uVision, add the path to CubeMX to the system path environment variable. The start uVision5 and create a project for an STM407ZG.

Start CubeMX by clicking the "play" button right of the STM32Cube Framework (API).

Now, the Olimex E407 board is equipped with a 12 MHz external crystal clock and a 32.768 kHz external RTC crystal, so we have to enable those pins first.

Switch to the "Clock Configuration" tab and input 12 MHz as HSE input frequency and then make the following selections:

Select HSE as PLL Soure Mux
Set the /M divider to /7
Set the *N multiplier to x196
Set the /P divider to /2
Select PLLSLK as System Clock Mux
Set APB1 Prescaler to /4
Set APB2 Prescaler to /2

Save it and select "Generate Code" from the Project menu.

When done, choose to close the CubeMX project and update the settings in uVision5.

A main.c file is generated for you that uses these new settings , ready to be filled in with your applicaiton source.

Good Luck

A Comment on Comments

2015-04-24T01:22:00.000-07:00

During recent years I have worked with many very tech savvy people worthy of a lot of respect for their insight in embedded development. However, the discussion regarding comments seems to never die. The more tech oriented the developers, the less they seem to like comments in source and the more they seem to believe self commenting code exist. It doesn't. What exist is self commenting thoughts in your head. I have no chance whatsoever to get a good grasp of your ideas without reading them, or asking you.

I have always taken a more customer focused approach, hoping that I won't have to explain myself too many times when I hand over the result of my efforts. Here are some things I would like to point out to all of you that doubt the usefulness code comments:

Code in itself can only describe what it does. Everything else are left out. All the experience you gained through experiments, fixing bugs and working with the environment is lost if you don't document it.
Code can't explain what it doesn't do. All the alternatives to the approach you have taken. All the quirkyness of the libraries you are using. That takes a long time to get your head around as a new employee or contractor.
Code doesn't explain its purpose. In best case, you can read out something from the function name or similar, but most of the time names like openAuxChannel leave you wandering WHY you would like to open an auxiliary channel.

There are lots of other reasons to write and maintain the comments in the source. But there is also the question of the cost for commenting code. If you comment your code you spend time that not feel very productive. In one sense that's correct, you don't write code, so you're not moving towards the goal of the project. But quality has to be considered too, and maintainability and tracability are such parameters. Without them, the quality of your code is poor. It eventually boils down to the cost of developing a feature vs. the cost of owning it. Over time, the cost of owning poor code is MUCH higher than the gain of developing poor code.

So, in the trade-off of commenting source and writing new features, I would like to promote some techniques that I practice:

Use doxygen! You might not like browse the source in rendered HTML or PDF documents, but it gives others a great overview and a starting point that lets them get get up to speed with the source quickly.
Write commenting scripts that puts skeletons of comments in your files, classes, members and functions. The cost of developing such a script is a lot less than the effort it saves. It takes away a lot of boring work as well! Often the IDE has some functionality for this, but I often get more fields into my doxygen comments with a script.
Don't comment getters and setters. Programmers are not stupid, they will understand anyway. Though, if you have a script putting doxygen comment on them, they will become browsable when the documentation is rendered, and that's very convenient.
Make sure you add comments to algorithmic code in the functions. These sections of comments are the most valuable. The not only explain your algorithm, it will affect your design too. If you notice that you have to explain flags and states set in objects or structures in other modules etc. it shows that your approach to the problem is probably not coherent enought, or easy enough. If you can't explain what you have done, you should refactor it.

Making Jersey work with Google App Engine

2012-10-18T12:30:00.001-07:00

I was looking for a way to get Jersey work in a Google App Engine application that I am working with. Things seems to work just fine with the local test server, but deployed in the production environment it fails miserably.

The answer to the problem wasn't easy to find at all, but after plenty of googling i found a this post that the answer I was looking for.

I will here summarize the steps in a bit shorter post and hope that it will make this solution a little bit easier to find for others.

The problem boils down to the dependency of the asm jar file. The new versions of the GAE libraries use asm-4.0.jar but Jersey still depends on asm-3.1.jar. There are some posts out there that suggest that you can remove the asm-4.0.jar and change the data nucleus version to v1. WARNING!!! Doing this messed my eclipse up totally. I never got it to work properly again and had to reinstall it and all the plug-ins!

What Harry Wye suggests on his blog is to repackage the jar files depending on asm-3.1 using Jarjar Links. This worked fine for me using version 1.3. 1.4 gave me a manifest attribute error. I am not terribly experienced in Java, so I don't know why this happend or what to do about it (if you know... please share it in a comment on this page), but trying 1.3 did the work for me.

To repackage the jersey server jar (which is the only one depending on asm), you first have to create a file (rule.txt) with this content:
rule org.objectweb.asm.** org.objectweb.asm3.@1

Then run jarjar:

java -jar jarjar-1.3.jar process testrules.txt asm-3.1.jar asm-3.1r.jarjava -jar jarjar-1.3.jar process testrules.txt jersey-server-1.14.jar jersey-server-1.14r.jar

Now you can deploy Jersey with the modified jars along with asm-4.0.jar in your GAE application.

Good Luck

DIY RS232 shield for the Olimex E407

2012-10-01T02:49:00.000-07:00

My first hardware project with the E407 was to create an RS232 shield (they can be bought if you don't want to build it yourself). I used olimed own arduino proto shield. I happened to have a Maxim 232 driver circuit in my drawer, and also the five 1uF electrolytic capacitors needed to create this board.

First of all I discovered that the Olimex proto shield doesn't fit the board very well. I had to grind off one of the corners to make it fit with the power supply capacitors on the E407 board. A regular proto shield won't fit better... it seems like the capacitors aren't too well placed on the E407 board :-/

Since I am not very experienced in programming the STM32 processor I had to verify the construction with an AVR I had laying around and it seemed to work fine. I couldn't make it work with the E407 though.

I found out that the source I had downloaded from ST's site is designed for the Discovery board which has an 8 MHz external clock source. The E407 has a 12MHz clock. I had to generate a new system file with ST's excel tool. The patch can be found here.

I then compiled the example from Elias electronics blog, which seemed to work fine.

Good luck!

Lean electronic prototyping: Prove your assumptions

2012-09-18T00:51:00.000-07:00

Conventionally, when developing electronic devices, companies tend to focus on developing a prototype hardware as the first stage. This is a complicated and expensive strategy that consumes large quantities of energy and money in a design up-front.

A complete electronics development studio, open source only!

There are alternatives however. Today there are many open source circuit boards to choose from off the shelf, that suits almost any need for an embedded system. These boards are extremely cheap, and the tools for them are almost always open source, and thus, also free.

The maker movement and DIY (Do It Yourself) trend has brought embedded development to be a serious hobby alternative for anyone... even kids. And there isn't just toy functionality offered on these cheap hobbyist boards, it is real computer power and connectivity there, such as USB, Bluetooth, internet working, motor control. There are even circuit boards meant to be sewn into your clothes with conducting thread! All from the price of 20$ USD or so. In combination with mobile phones, cloud services and integration with other systems, these systems form a very powerful tool for innovation.

One of the most eye catching applications is off course the home 3D printers that has started to pop up in the market. For about 600$ USD, you get your own mechanical factory! The CAD-tools used for these are also often free and/or open source. With the 3D printers you can manufacture mechanical parts for your invention with very small means. This becomes a very important missing piece in the prototyping process that often tends to become the most expensive step when developing the prototype.

With the right prototyping strategy, a company can focus on developing quite advanced Minimum Viable Products that they actually can test in the hands of a real customer. This will give a much more trust worthy feedback, and will help the company to eliminate waste in the development process.

The sad truth is that the most important properties of a system are the least costly to develop. Many electronic devices are completely depending on a good the user experience, which in most cases are implemented in software. I'm not saying that the industrial design of a device is unimportant, on the contrary, but in the development of a minimum viable product, or prototype, you should focus on proving your assumptions regarding the customers needs/desires. To do this, the shortest and cheapest way is often to realize the solution in software. With the use of a lean prototyping strategy, the trade off between software functionality and industrial design can become less dramatic.

So what's the catch? ...

Well, there are off course down sides of these methods. To face the truth, the quality of the open source tools often leave a lot to wish for. Setting up a development environment based on open source CAN be done, but it takes it's fair amount of googling and experimenting to get it running. The documentation is often in a poor state, or none existing. This puts a big burden on the developers involved and the project becomes a lot more dependent on a high level of skills of the team members.

I would like to point out though, that during my time as a devloper and project manager in embedded systems projects, I cannot recall one single occasion when I have received any useful support from a company we have bought "closed source" tools from. So the increase in risk, is probably leveled out by the probability that you would actually get what you pay for, when using proprietary closed source software.

So keep it lean; focus on the customer, eliminate waste, keep the iterations short.

Building the GCC tool chain for the STM32F4

2012-09-15T04:21:00.000-07:00

So... I have to develop some applications on bare metal arm processors and need to practice a bit on a development board before I get started with the customer. The applications will be written in C, but I also want to investigate how C++ fits a project like this. The program is an audio application which eventually will use some filtering and asynchrounous USB transfer.

I have chosen an Olimex Board (STM32-E407) since it conforms to the Arduino form factor and has a processor close to the model used by my custormer. A JTAG was and a couple of development shields were ordered as well.

Before I can get started I need a toolchain to compile my applications. I could download a prebuilt toolchain from Mentor Graphics (Code Sourcery), but I wanted to see how if I could build one myself.

I have a 32 bit Ubuntu 12.04 host, and I tried to follow two other howto’s but none of them worked, so I had to mix the instructions from both. (http://www.triplespark.net/elec/pdev/arm/stm32.html, http://jeremyherbert.net/get/stm32f4_getting_started )

First install the tools needed for building the toolchain:
$sudo apt-get install flex bison libgmp3-dev libmpfr-dev libncurses5-dev libmpc-dev autoconf texinfo build-essential libftdi-dev libzip-dev

Now clone the source repository:
Note: See the update below for instructions for GDB XML support!
$git clone git://github.com/esden/summon-arm-toolchain
$cd summon-arm-toolchain

Since noone else will use my toolchain I will install the toolchain in my home directory: ~/tools/arm/arm-linaro-eabi-4.6.

Build the tool chain:
$./summon-arm-toolchain PREFIX=~/tools/arm/arm-linaro-eabi-4.6/ OOCD_EN=1 TARGET=arm-none-eabi LIBSTM32_EN=1 USE_LINARO=1 CPUS=2

When done, check the result:
$/bin/arm-none-eabi-gcc --version

It should say something like this:
arm-none-eabi-gcc (Linaro GCC 4.6-2011.10) 4.6.2 20111004 (prerelease)
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

To be able to work efficiently on the command line I create an environment setup script. With this script I can have several different toolchains for different platforms, and different automation scripts, that doesn't pollute my systems PATH environment variables and bash setup.

Open up a text editor and create the file env.sh, looking like this:

#!/bin/sh
echo "Olimex STM32-E407 Development Environment"

(
#Sub shell
#Export the variables to use in the development environment
export PATH=~/tools/arm/arm-linaro-eabi-4.6/bin:$PATH
export PS1="[Olimex E407] \u@\h:\w\$ "

#Start a new shell with the exported variables
/bin/sh
)

Change the mode of the file:
$chmod +x env.sh

Enter your environment:
$./env.sh

To exit the environment shell, just type exit and you’ll be back to your normal prompt.

From here I followed the last sections in http://jeremyherbert.net/get/stm32f4_getting_started to build the first test software.

Good luck!

Update Sept. 17 2012

To be able to debug the target I hade to rebuild the toolchain with XML support enabled in the GDB target.

To do this you first need to install expat:

$sudo apt-get install libexpat1 libexpat1-dev

Then find the GDB build target in the automated build script named summon-arm-toolchain and add the parameter "--with-expat \" before the GDBFLAGS parameter (or add it to the GDBFLAGS variable).

Now build the tool chain according to the instructions above.

Setting up Olimex arm-usb-tiny-h with Ubuntu 12.04

2012-09-14T15:31:00.000-07:00

Along with the Olimex E407 board, I also ordered an Olimex arm usb tiny h JTAG to be able to program the board. I am more convenient with working in Linux than other operating systems so I really wanted this to work with my Ubuntu 12.04 (32bit x86) installation.

This was a bit tricky, and it has taken some time to Google for all the bits and pieces that finally made it work.

First I downloaded libftdi 0.20 from here: http://www.intra2net.com/en/developer/libftdi/download.php
Build it with the normal series of auto tools commands:

$./configure
$make
$sudo make install

I am not sure if both are needed, but to be on the safe side I downloaded the ftd2xx driver and installed it as well. You can get it here: http://www.ftdichip.com/Drivers/D2XX.htm. Copy the libftd2xx.so.1.1.12 and libftd2xx.a to /usr/local/lib and make symlinks with the names libftd2xx.so and libftd2xx.so.1

I downloaded and compiled openOCD 0.6.0 from here http://openocd.sourceforge.net/ and build it with:

Note: See updated instruction regarding patch below!
$./configure --enable-ftdi --enable-ft2232_ftd2xx
$make
$make install

When plugging in the arm-usb-tiny-h, you will not be granted access by default. To handle this you have to create a rule for udev:

$cd /etc/udev/rules.d

As root, create a file named 99-olimex-arm-usb-tiny-h.rule with this content:

SUBSYSTEMS=="usb", ATTRS{idVendor}=="15ba", ATTRS{idProduct}=="002a", MODE="0666"

I also removed the brltty package which might interfere with some of the ftdi drivers, but this was only a precaution. You could try running without removing it.

$sudo apt-get remove brltty

Now I run openOCD with the following command (paths relative from where I built openOCD):

$openOCD -f tcl/interfaces/ftdi/olimex-arm-usb-tiny-h.cfg -f stm32f4x.cfg

You can download the stm32f4x.cfg from Olimex web site.

Good luck!

Update Sept 17 2012
Apply the patch provided by this site: http://pastebin.com/rdFF2eZd before configuring and buildng openOCD

Ssh-agent on Linux Mint 12

2012-04-02T00:04:00.000-07:00

I have had som struggle setting up ssh agent to work with my Linux Mint machines. Both are running Linux Mint 12 and are fairly recently installed. What I want to do is to be able to log in from one host (called vader) on the other (called leia) without having to enter password every time and I don't want to generate keys with empty passwords.

First of all, I discovered that on Linux Mint 12, the firewall is turned on by default which prevented me from loging on to 'leia' with ssh. So, make sure your firewall is letting traffic through on port 22.

Now, generate keys as usual on your local host and enter a password for them:

vader$> ssh-keygen -t rsa

Install the public key on the host you want to connect to:
ssh-copy-id -i @

In my case, it looks like this:
vader$> ssh-copy-id -i ~/.ssh/id_rsa.pub peter@leia

Now, it is time to load your key into memory. This is done with the program ssh-add, but first you have to start an instance of ssh-agent.
vader$> ssh-agent bash
vader$> ssh-add

The ssh-add command will automatically load ~/.ssh/id_rsa into the agent and ask you for the password you've set. From now on I don't have to enter a password when logging in to leia, as long as I stay in the ssh-agent shell.

Good luck!

Windows Temp folder prevents Eclipse from starting

2012-03-23T01:24:00.001-07:00

Since a couple of months I have had the pleasure of working on a Google Web Toolkit (GWT) project. Google have produced a range of quite well integrated tools for it an it is quite a smooth experience, even on an embedded platform such as the one I am working with.

Unfortunatelly, it seems that the Google Plugin for Eclipse (GPE) relies heavily on the users local Temp folder in windows, and generates quite a lot of files to it. After a couple of months, first eclipse, then MinGW, simply wouldn't start. Or sometimes i took hours to get it working. It turns out that you can't have too many files and folders in a folder, and there is no mechanism in Windows (that I know of) to automatically clean out the Temp folder or even give a warning when it becomes full. If the Temp folder gets full, it prevents some programs from operating correctly.

Solution: Delete the content of the Temp folder ("C:\Users\'your user name'\Appdata\Local\Temp" in Windows 7). It's as simple as that.... and it worked like a charm for me.

Good Luck

Hp 3210 URL for scanning

2011-03-25T00:05:00.000-07:00

I have a Hp 3210 printer/scanner/copier. It works perfect with Ubuntu, but everytime I have to set it up for scanning i run into problems finding out what parameters I need to start xsane. Here's how to find the url:

Use the program hp-makeuri to generate the uri

$ hp-makeuri 192.168.1.20

the result is in my case:

SANE URI: hpaio:/net/Photosmart_3200_series?ip=192.168.1.20

Now xsane can be started with the uri as parameter:

xsane hpaio:/net/Photosmart_3200_series?ip=192.168.1.20

xmlHttpRequest.status=0

2011-03-24T03:42:00.000-07:00

Hi everyone!
This will be a short one, but I just came across some problems during the process of learning javascript again. It's a long time since I worked with client side web technology, so it's fun to discover how far things have come during recent years.

Anyway, in my effort of building a GMail Contextual gadget i have to send some data to a http server using xmlHttpRequest. The gadget runs in an iFrame which will stop you from using xmlHttpRequest. It's called cross domain blocking or something. So the first you'll have to do is to short circuit the cross domain blocking by adding the header "Access-Control-Allow-Origin: *" to the response.

In php:

header('Content-type: text/html');
header('Access-Control-Allow-Origin: *');
...

If you don't do this you will inevitably get redystate=4 and status=0 (or 'unknown' or something) for the xmlHttpRequest object.

The second mistake i made was the trigger button for the request. Using a submit input and the onClick event will generate two callbacks. One to the forms action, and one to the actual function you specified in the onClick. This messes things up and you're back with status=0 problem again.

Solution: Use input type="button". This input will not trigger the forms actionl.

Example:

<form name="fakeform" action="return false;" >
<input type="button" value="Go" onClick="doRequest();" />
</form>

Here is a good example of how to write the doRequest function: http://stackoverflow.com/questions/247483/http-get-request-in-javascript

Good luck!

Google Sites

2009-07-09T00:41:00.000-07:00

I mina försök att tränga djupare i Googles tekniker och produkter har turen kommit till Google Sites, en onlinetjänst för att skapa egna websidor. Inte så dumt faktiskt... MEN... Inte så färdigt och integrerat med andra googleprodukter som man kunde hoppats.

Google sites är enkelt att använda och man kommer snabbt in i hur man lägger till s.k. gadgets i sidan. Det är precis samma gadgets som används på iGoogle. Så långt allt bra. Vad jag skulle vilja se mer av är bättre integration av Google Groups, Blogger och liknande produkter. Det skulle ge en bättre möjlighet att skapa riktiga "portaler" för sina egna projekt och liknande.

När man granskar videos kring Sites verkar det helt klart vara möjligt att skapa den sortens innehåll, och det kan ju vara jag som missat nåt, men då kan man väl konstatera att det är för svårt att göra innehållet riktigt bra.

RSS

2009-07-06T05:46:00.000-07:00

Igentligen är det här inget annat än ett försök att upptäcka nya tekniker. Det kommer ju mycket nu kan man säga. I mitt yrke som utvecklare inom "embedded systems" blir man lätt lite insnöad på elektroniken, och då är det lätt att hetare tekniker drar ifrån.

Det hela började med att jag experimenterade en del med RSS-feeds. Feeds är ju som sagt inte något nytt, men jag har inte riktigt intresserat mig för deras existens mer än att jag lagt till en eller annan i min browser för att följa nyheter där. Det är väl gott och väl. Men så provade jag att lägga till feeds i outlook, och se vilken skillnad det blev!!!

Helt seriöst... jag trodde nog inte att rss-läsarens utformning skulle ha så stor inverkan på informationsflödet. Effektivitetsskillnaden i att kunna se vilka feeds man läst och inte är enorm. Jag tänkte genast att det här kanske är en bra metod att använda i utvecklingsgruppen på jobbet, så jag slängde upp en RSS-kanal på samma webserver som vi har vår wiki och länkar nu alla feeds till sidor på wikin. Det funkar huur bra som helst!

Nu slipper medlemmarna i min grupp aktivt gå in och kolla om det finns uppdateringar de borde ta del av på wikin, det kommer de se i outlook! Tyvärr saknar jag något bra linuxbaserat verktyg för att publicera rss-kanalen. Gör det manuellt än så länge.