My Computer Adventures

FxGqlC Chocolatey package

2016-02-10T20:43:00.000+01:00

Good News! FxGqlC is now also accepted as official Chocolatey package.
So, from now on, the easiest way to install FxGqlC on Windows is:

Open a PowerShell window as administrator
If you're not already using Chocolatey, install it by running:
iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))
Install FxGqlC by running:
choco install fxgqlc

This post has been moved to http://mycomputeradventures.foxinnovations.be/2016/02/10/fxgqlc-chocolatey/ .

Amazon Glacier pricing example - Updated with latest price reduction

2015-11-22T20:00:00.000+01:00

Let's say you want to backup 100 GB (= +/- 25 DVDs), e.g. 1000 files of 100 MB.

(Prices for e.g. Ireland are 10% higher than the ones used in this example)

To back up your data:

Backup:
To back up your data

Incoming Transfers: free
Upload requests: 1000 x (0.050$ / 1000 requests) = 0.05$ is virtually free
Total Backup cost: 0.05$, so virtually free

Storage:
To keep your data at Amazon Glacier

100 GB x (0.007$ / GB / month) = 0.7$ / month
Total storage cost: 8.4$ / year

Restore: (let's say you want to download 10GB of 100GB, spread over a month: 3 files per day)

Restore: 5% of 100GB free, remainder:
5GB x 0.010$ = 0.05$
Outgoing Transfers: 1GB for free, remainder:
9 GB x 0.09$ = 0.81$
Requests: 100 x 0.05$ / 1000 requests = 0.005$
Total restore cost = 0.87$

And for 1 TB of data:

Backup: 0.50$
Storage: 10$/month or 84$/year
Restore of 10%: 9.46$

It doesn't matter if you use Amazon Glacier directly, or via the "Glacier storage class" in Amazon S3. In the latter case, the Glacier costs are advertised under S3.

To have the cheapest costs, you should optimize between:

The number of objects: LESS objects is BETTER, since many objects cost more than less objects (both for backup and restore).
E.g. if you use files of 100 KB instead of 100 MB, the cost for backup is 50$ instead of 0.05$.
The size of the objects: SMALLER objects is BETTER. To be able to spread the download over a month, you should keep the size of the objects below 1/30th of 5% of your total size. So, at least 600 files if they are equal in size.
E.g. if you use 10 files of 10 GB instead of 100MB, the cost for restoring 10% (1 file) is 0.78$ instead of 0.05$

So, the only thing you need to prevent is storing many small objects on Glacier.

Some more remarks:

The easiest way to get started with Glacier, is using the Glacier storage class in Amazon S3. This gives you an easy to use GUI to store and retrieve files to/from Glacier, with a small extra temporary S3 cost. Transfers from S3 to Glacier are free of charge, so the biggest extra cost is the storage cost while the data is on S3 instead of Glacier.
For a restore, you need to be patient: it takes 4 hours until your data is available. Not instantly available, but also 10 times cheaper than S3.
And the fault tolerance is remarkable: 11 nines (99.999999999%). This is huge. Especially compared with portable harddisks (99%?) or DVDs (99.999%?)...
The cost of certain datacenters (e.g. Frankfurt) is higher. Ireland has now the same costs as the US locations.

This post has been moved to http://mycomputeradventures.foxinnovations.be/2015/11/22/amazon-glacier-pricing-example/ .

Download pictures from Picasa Webalbums

2015-11-03T21:50:00.001+01:00

A simple but very useful tool to download one or all (public) Picasa Web Albums of a user. Especially since it is still not clear what the future of Picasa Web Albums will be... Download on Github:
https://github.com/WimObiwan/FxPicasaWebAlbumDownloader/releases/latest

This post has been moved to http://mycomputeradventures.foxinnovations.be/2015/11/03/download-pictures-from-picasa-webalbums/ .

Visual C++ Code Analysis fatal error "Compiler is out of heap space"

2015-08-11T20:00:00.000+02:00

When running Visual C++ code analysis (in the Visual Studio IDE or command line) on a big project, you can get this fatal error: C1060 compiler is out of heap space. E.g.:

Running Code Analysis for C/C++...
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\locale(70): fatal error C1060: compiler is out of heap space [C:\YourPath\YourProject.vcxproj]
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\locale(65) : while compiling class template member function 'size_t std::collate::_Getcat(const std::locale::facet **,const std::locale *)'
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\xlocale(607) : see reference to function template instantiation 'size_t std::collate::_Getcat(const std::locale::facet **,const std::locale *)' being compiled
          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\locale(140) : see reference to class template instantiation 'std::collate' being compiled

On a 64-bit machine, the solution is to run the code analysis from the command line using the 64-bit version of MSBuild.exe:

"C:\Program Files (x86)\MSBuild\12.0\Bin\amd64\MSBuild.exe" /p:Configuration=Release /p:Platform="x64" /p:RunCodeAnalysis=true "C:\YourPath\YourProject.vcxproj"

So, use the one in the subdirectory

amd64

. You need to replace the configuration/platform/project according to your needs. By default, even when releasing 64-bit executables, the 32-bit version of the compiler is used. This error was also the reason that certain compilations were hanging on our Jenkins/Hudson build servers: after this error, the MSBuild.exe process never terminated until it was killed (in Windows Task Manager) or aborted (in the Jenkins Web GUI.

This post has been moved to http://mycomputeradventures.foxinnovations.be/2015/08/11/visual-cpp-code-analysis-out-of-heap-space/ .

Extracting data from BTrieve files

2015-08-03T21:57:00.000+02:00

The procedure below works with BTrieve 6.15, but probably works with older and more recent releases of BTrieve. Even for PervasiveSQL deployments that - for legacy reasons - use BTrieve as their backend, it should work. But in the latter case, you should use the SQL interface (ODBC connector).

First, you need to extract the binary information from the BTrieve file.
- Download:
  BTrieveFileSaver.zip
- Run:
  BtrieveFileSaver.exe -btrin MYTABLE.BTV -btrout c:\temp\MyTable.txt
The BTrieve file only contains the data, and no information about the model.

When you have the source, that reads or writes the data, then you can extract the model.
e.g.:

             typedef struct _MyRecord
 OFF  LEN    {
   4    4       long  id;
   8   32       char  name[32];
 (40)        } MyRecord;

The lengths are derived from the datatypes (C++ in this case). The cumulative lenths are determine the offset of each field. For some reason, there is an initial offset of 4 bytes.
If you don't have the source code, you can possibly "reverse engineer" the data from the table.

Using a PowerShell script, you can extract, filter and format the data. In this case, we extract the id and name fields of all records, and show them as a table.

A helper function to make it easier to parse the strings

Add-Type -AssemblyName "System.Text.Encoding"
 
function GetString([byte[]] $buffer, [int] $start, [int] $maxlen) {
   $i = $start
   for ($i = $start; ($i -lt $buffer.Length) -and ($i -lt ($start + $maxlen)); $i++) {
      if ($buffer[$i] -eq 0) {
         break
      }
   }

   $len = $i - $start
   [System.Text.Encoding]::ASCII.GetString($buffer, $start, $len)
}

The actual extraction is simple:

$lines = Get-Content c:\temp\MyTable.txt
$table = $lines | foreach {
   $line = $_
 
   $id = [bitconverter]::ToInt32($line, 16)
   $name = GetString $line 26 48
 
   $obj = New-Object psobject
   $obj | Add-Member -MemberType NoteProperty -Name 'id' -Value $id
   $obj | Add-Member -MemberType NoteProperty -Name 'name' -Value $name
   $obj
}
 
$table | Where-Object { $_.name-ne '' } | Sort-Object id | Format-Table -AutoSize

This post has been moved to http://mycomputeradventures.foxinnovations.be/2015/08/03/extracting-btrieve-data/ .

Running Visual Studio code analysis in Jenkins/Hudson

2015-08-02T20:00:00.000+02:00

Code analysis can be extremely useful to detect certain errors that are difficult to find. One example is errors in the 'printf' format specifier, e.g.

int iValue = 17; 
printf("%s", iValue);

Since code analysis takes a long time, it is a good example of something that could be done on the build server, e.g. every night.
First make sure that the PowerShell plugin is installed.

Create a new project, add a Powershell build step, and paste this script:

$sln='.\MySolution.sln'    # path based on the workspace root
$configuration='/p:Configuration=Release'
$platform='/p:Platform="Win32"'

#only take these warnings into account
$severe_warnings_list = @(
    4789 #: Destination of memory copy is too small
    6053 #: Call to function may not zero-terminate string
    6057 #: Buffer overrun due to number of characters/bytes mismatch
    6059 #: Incorrect length parameter
    6063 #: Missing string argument
    6064 #: Missing integer argument
    6066 #: Non-pointer passed as parameter when pointer is required
    6067 #: Parameter in call must be the address of the string
    6066 #: Non-pointer passed as parameter when pointer is required
    6209 #: Using sizeof when a character count might be needed.
    6269 #: Possible incorrect order of operations: dereference ignored
    6270 #: Missing float argument to varargs function
    6271 #: Extra argument: parameter not used by the format string
    6272 #: Non-float passed as argument  when float is required
    6273 #: Non-integer passed as a parameter when integer is required
    6278 #: array new [] paired with scalar delete
    6281 #: relational/bitwise operator precedence problem
    6282 #: Incorrect operator: assignment of constant in Boolean context
    6283 #: array new [] paired with scalar delete
    6284 #: Object passed as a parameter when string is required
    6290 #: Bitwise operation on logical result: precedence error
    6302 #: Format string mismatch: char* when wchar_t* needed
    6303 #: Format string mismatch:  wchar_t* when char* needed
    6306 #: Incorrect call to 'fprintf*': consider using 'vfprintf*'
    6328 #: Wrong parameter type passed
    6334 #: Sizeof operator applied to an expression with an operator
    6336 #: Arithmetic/question operator precedence problem
    6522 #: Invalid size specification: expression must be of integral type
    6523 #: "Invalid size specification: parameter 'size' not found
)

$a = $(C:\PROGRA~2\MSBuild\12.0\Bin\MSBuild.exe $configuration $platform /p:RunCodeAnalysis=true $sln)

# Take out certain link and BSC compiler warnings
$b = $a | where { $_.Contains('warning') -and $_ -match '^  ' -and $_ -notmatch 'warning LNK' -and $_ -notmatch 'warning BK' }
# Take out 3rd-party libraries
$b = $b | where { ($_ -replace ('^\s*' + [regex]::Escape($env:WORKSPACE) + '(.*?)\(\d+\)\: .*$'), '$1') -notmatch '^\\libraries\\(?:boost|x(?:un)?zip\.cpp|xmlite\.cpp)' }
# Only keep warnings from the severe_warnings_list
$severe = @($b | where { ($_ -replace '^.*\: warning C(\d+)\:.*$', '$1') -in $severe_warnings_list })

# Show the warnings
$severe

if ($severe.Length -gt 0) { 
    Write-Warning "Found $($severe.Length) severe code analysis warnings!"
    exit 1
} else {
    ($b -replace '^.*\: warning (C\d+)\:.*$', '$1') | group | sort -Descending Count
}

exit 0

This post has been moved to https://mycomputeradventures.foxinnovations.be/2015/08/02/running-visual-studio-code-analysis-in-jenkins-hudson/ .

TomTom very slow GPS lock

2015-07-14T22:53:00.002+02:00

During the last few weeks, my TomTom Live 825M Europe was extremely slow in finding the GPS location. Sometimes more than 15 minutes, which is totally unacceptable. TomTom support proposed to bring the device up-to-date, which I already did without better results.

The "drum roll reset" as proposed on a few forums as a solution for other issues, solved all problems! Hold the power button for up to 30 seconds until you here the drum roll. A GPS location lock is now achieved in a few seconds after power on.

This post has been moved to http://mycomputeradventures.foxinnovations.be/2015/07/14/tomtom-very-slow-gps-lock/ .

Copying symbolic links as regular files

2015-07-12T22:41:00.000+02:00

By default, symbolic links (symlinks) are copyied as they are. If you want to copy the destination file of the symlink ("follow" the symlink), you need to add the argument "-L".
For example:

cp -L /mnt/data/archive/Documents /mnt/backup

This is necessary when copying files from a git-annex repository, which uses symlinks to an internal filestore.

A good explanation between hard and soft links on linux can be found here:
http://linuxg.net/the-linux-and-unix-links-the-symbolic-link-vs-the-hard-link/

This post has been moved to http://mycomputeradventures.foxinnovations.be/2015/07/12/copying-symbolic-links-as-regular-files/ .

Dependency on Visual Basic 5 of legacy applications

2015-07-11T22:31:00.000+02:00

It sounds impossible, but there are still applications distributed that need the runtime of Visual Basic 5... (released Februari 1997) The runtime files are not available in recent versions of Windows, so you get an error like this:

This application failed to start because msvbvm50.dll was not found. Re-installing the application may fix this problem.

The solution is simple: download and run this installer from Microsoft:

http://download.microsoft.com/download/vb50pro/utility/1/win98/EN-US/Msvbvm50.exe

More information can be found on this knowledge base article:

https://support.microsoft.com/en-us/kb/180071

Originally, I encountered the problem when starting a Flemish/Dutch CD from the kids, called "Vakantieblaadjes":

Deze toepassing kan niet worden gestart omdat msvbvm50.dll niet is gevonden. Opnieuw installeren van de toepassing kan dit probleem oplossen.

Jenkins GIT polling blocked

2013-07-05T22:08:00.003+02:00

We were experiencing this error on our Windows Hudson/Jenkins build servers multiple times a week:
There are more SCM polling activities scheduled than handled, so the threads are not keeping up with the demands. Check if your polling is hanging, and/or increase the number of threads if necessary.

We are using GIT, and apparently, the cause of the blocked GIT pollings was a hanging SSH.exe process.

A few posts indicated that others are experiencing the same or similar problems:
Jenkins-14046: git operations occasionally hang in Jenkins on Windows
StackOverflow: git operations occasionally hang in Jenkins on Windows

The proposed solutions didn't help me:
* Reinstalled Jenkins and GIT.

* Installed plink.exe/pageant.exe, and set GIT_SSH.
* Played around with the temporary directory.

As a workaround, the only thing I could to make everything run again (besides rebooting) was killing the blocked ssh.exe process. So currently, I'm running a Windows Scheduled task that kills blocked SSH.exe processes every 5 minutes:

Get-Process | where { $_.ProcessName -eq 'SSH' -and $_.StartTime -lt (Get-Date).AddMinutes(-2) } | Stop-Process -Force

Configuration of the Windows Scheduled task:

As argument, you can specify:

-Command "& { Get-Process | where { $_.ProcessName -eq 'SSH' -and $_.StartTime -lt (Get-Date).AddMinutes(-2) } | Stop-Process -Force }"

With Powershell 3 (as administrator), you can use the built-in support for ScheduledJobs:
$trigger = New-JobTrigger -At (get-date) -Once -RepetitionInterval (New-TimeSpan -Minutes 5) -RepetitionDuration ([TimeSpan]::MaxValue)
Register-ScheduledJob -Name Jenkins-Workaround-BlockedSsh -ScriptBlock { Get-Process | where { $_.ProcessName -eq 'SSH' -and $_.StartTime -lt (Get-Date).AddMinutes(-2) } | Stop-Process -Force } -Trigger $trigger
# that's all you need to do!

Output:
Id         Name            JobTriggers     Command                                  Enabled
--         ----            -----------     -------                                  -------
40         Jenkins-Work... 1                Get-Process | where { $_.ProcessName... True

Get list of all ScheduledJobs:

Get-ScheduledJob *   # gets a list of all jobs

Output:
Id         Name            JobTriggers     Command                                  Enabled
--         ----            -----------     -------                                  -------
...
40         Jenkins-Work... 1                Get-Process | where { $_.ProcessName... True

Get properties of the newly created ScheduledJob:
Get-ScheduledJob -Id 40 | Format-Custom -Depth 1  # replace id 40 with the correct one
class ScheduledJobDefinition
{
  ...
  Definition =
    class JobDefinition
    {
      Name = Jenkins-Workaround-BlockedSsh
      JobSourceAdapterType = Microsoft.PowerShell.ScheduledJob.ScheduledJobSourceAdapter
      ModuleName = PSScheduledJob
      JobSourceAdapterTypeName = ScheduledJobSourceAdapter
      Command =  Get-Process | where { $_.ProcessName -eq 'SSH' -and $_.StartTime -lt (Get-Date).AddMinutes(-2) } | Sto
      p-Process -Force
      InstanceId = dd6cfd1d-5cc1-4ecc-8172-93f09ff0beda
      CommandInfo =
    }
  Options =
    class ScheduledJobOptions
    {
      StartIfOnBatteries = False
      StopIfGoingOnBatteries = True
      WakeToRun = False
      StartIfNotIdle = True
      StopIfGoingOffIdle = False
      RestartOnIdleResume = False
      IdleDuration = 00:10:00
      IdleTimeout = 01:00:00
      ShowInTaskScheduler = True
      RunElevated = False
      RunWithoutNetwork = True
      DoNotAllowDemandStart = False
      MultipleInstancePolicy = IgnoreNew
      JobDefinition = Microsoft.PowerShell.ScheduledJob.ScheduledJobDefinition
    }
  Credential =
  JobTriggers =
    [
      Microsoft.PowerShell.ScheduledJob.ScheduledJobTrigger
    ]
  Id = 40
  GlobalId =
    class Guid
    {
      Guid = 22ef5ef5-06b6-41de-99dc-6fc5b6fee6b6
    }
  Name = Jenkins-Workaround-BlockedSsh
  Command =  Get-Process | where { $_.ProcessName -eq 'SSH' -and $_.StartTime -lt (Get-Date).AddMinutes(-2) } | Stop-Pr
  ocess -Force
  ExecutionHistoryLength = 32
  Enabled = True
  ...
}

After about 5 minutes, the first Job will be created: 
Get-Job | where { $_.Name -match 'Jenkins' }
Get-Job -Id 481 | Format-Custom -Depth 1

This post has been moved to http://mycomputeradventures.foxinnovations.be/2013/07/05/jenkins-git-polling-blocked/ .

GIT Merge conflict solving with kdiff3

2013-01-25T08:00:00.000+01:00

First we will create a test repository and a test file on which we will create a merge conflict later on.
(in this example, the commands are executed with PowerShell in Windows, but the GIT commands are exactly the same on any other shell or OS)

Creation of test repository
> cd \tempmd mergetestcd mergetest
> git init
Creation of test file
> 'Original line' | Out-File file.txt
Commit into repository
> git add file.txt
> git commit -m "Original checkin"
Current repository state: (using the tool GIT Extensions)

Now we will create a new branch A, and update the file.

Create new branch
> git branch branch-A
> git checkout branch-A
Update file
> 'Branch A line' | Out-File file.txt
Commit into repository
> git add file.txt
> git commit -m "Branch A change"
Current repository state:

Now we will create a new branch B, and update the file with a conflicting change.

Create new branch, starting from the original master branch
> git checkout master
> git branch branch-B
> git checkout branch-B
Update file
> 'Branch B line' | Out-File file.txt
Commit into repository
> git add file.txt
> git commit -m "Branch B change"
Current repository state:

All this was a preparation for what we will do now: merge the conflicting change of branch-B into branch-A.

Select the target branch (in this case branch A)
> git checkout branch-A
Merge the source branch (in this case branch B)
> git merge branch-B
This results in a merge conflict:
Auto-merging file.txt
CONFLICT (content): Merge conflict in file.txt
Automatic merge failed; fix conflicts and then commit the result.
This conflict can be resolved by starting your mergetool
> git mergetool
It gives you a message before it launches your mergetool (in my case kdiff3):
Merging:
file.txt

Normal merge conflict for 'file.txt':
{local}: modified file
{remote}: modified file
Hit return to start merge resolution tool (kdiff3):
You get a window containing 4 versions of the same file:

Top left: "Base". This is the original version of the file, the last shared version between what became later branch-A and branch-B.
Top Middle: "Local". This is the target branch version of the file. It is called "Local", because this is the currently selected branch (using the last git checkout command).
Top Right: "Remote". This is the source branch version of the file. It is called "Remote", as apposed to "Local".
Bottom: "Output". This is the merged version of the file. After successful completion of the merge, this will become the new version of the target branch. In this pane, you can edit the text, or you can right click on any merged part and select what version(s) of the original file you want to include in the output. In this case, I decide to include the changes done in both branch-A and branch-B.
Close the kdiff3 application (saving the file), and commit the merge:
> git commit
(accept the default commit message)
This brings us to the following repository state:
We can check if the content of the test-file is what we need:
> Get-Content file.txt
Branch A line
Branch B line

This post has been moved to http://mycomputeradventures.foxinnovations.be/2013/01/25/git-merge-conflict-solving-with-kdiff3/ .

FxGqlC: Added new function 'PREFIX'

2013-01-17T22:34:00.005+01:00

Added (non-aggregation) text function PREFIX to return the common prefix of two strings.
An aggregation function PREFIX (with 1 argument) was already added in v2.4.

SELECT PREFIX('0032478123456', '0032478654321')
-- returns '0032478'

This feature is added to FxGqlC in v2.5-alpha5.

FxGqlC: Added aggregation functions ENLIST and ENLISTDISTINCT

2013-01-17T22:34:00.002+01:00

Added aggregation functions ENLIST and ENLISTDISTINCT.
ENLIST creates a string value containing the list of all string values in their original order.
ENLISTDISTINCT creates a similar list, but the dupplicate values are removed, and the list is ordered.

SELECT [Winner], COUNT(*), ENLIST([Tournament])
   FROM ['SampleFiles/Tennis-ATP-2011.csv' -Heading=On]
   GROUP BY [Winner] ORDER BY 2 DESC
SELECT [Winner], COUNT(*), ENLISTDISTINCT([Tournament])
   FROM ['SampleFiles/Tennis-ATP-2011.csv' -Heading=On]
   GROUP BY [Winner] ORDER BY 2 DESC

This feature is added to FxGqlC in v2.5-alpha5.

Ubuntu Server distribution upgrade

2012-10-28T19:55:00.000+01:00

Doing a distribution upgrade is easy in the graphical user interface. On a "headless" Ubuntu Server, you probably don't have a graphical user interface available, and you need to perform a few easy steps.

First you need to check 2 things:

The package "update-manager-core" needs to be installed. Just run the following command: (it will install it, if it is not already)
sudo apt-get install update-manager-core
On Ubuntu Server, the update channel is usually "LTS" (Long Time Support) which will upgrade only to the major release of Ubuntu Server. Probably, you want to set this to "normal", to allow an upgrade to every distribution upgrade. To do this open the file "/etc/update-manager/release-upgrades", and change the line "Prompt=lts" to "Prompt=normal".
For example, with pico:
sudo pico /etc/update-manager/release-upgrades

Change the Prompt setting to "normal".

Press Ctrl-X, press "Y" to confirm and press enter to confirm the filename.

Then the distribution upgrade can be started by running this command:

sudo do-release-upgrade -d

It is not "recommended" to do a distribution upgrade remotely over ssh. The "do-release-upgrade" command above checks on this, and gives a warning. However, I didn't have any problem doing the upgrade remotely over ssh.

Converting a date/time to time_t using FxGqlC (or SQL)

2012-10-18T21:00:00.000+02:00

The time_t is used in C++ to represent a date/time. It is expressed in seconds since Januari 1st, 1970.

To get the current date/time as a time_t value, you can run this query in FxGqlC (or SQL):
select datediff(second, '1970-01-01', getutcdate())

You need to use getutcdate() because time_t defines the UTC time.

Or for any arbitrary date/time (in UTC):
select datediff(second, '1970-01-01', '2012-10-18 22:33')
-- Returns 1350599580

The other way around is also easy: run this query to convert a time_t to a date/time
select dateadd(second, 1234567890, '1970-01-01')
-- Returns 13/02/2009 23:31:30

LibreOffice doesn't appear in alt-tab view

2012-10-15T20:27:00.000+02:00

I have an annoying problem with Unity on Ubuntu: it often happens that LibreOffice/OpenOffice Calc or Writer windows don't appear in the alt-tab application switching.
Googling the problem learned me that it is a known problem: https://bugs.launchpad.net/bamf/+bug/1026426
The good news is that it is solved in Ubuntu 12.10, and a patch will be available for 12.04 LTS. A workaround is to restart Unity:

Start a command window (Ctrl-Alt-T)
Run:
unity --replace & disown

Python scripts inside PowerShell window

2012-09-05T20:00:00.000+02:00

After installing Python on Windows, you can start a Python script by executing the script file-name.
E.g.:
./MyScript.py

However, the command is not executed in the same PowerShell window. It opens a new command window to start "python.exe", which executes the command (and immediately closes the window).

This behavior can be changed, and the script can be executed within the current PowerShell window.

Just add the ".py" file extension to the PATHEXT environment variable. This can be done by executing this command:
$env:PATHEXT += ";.py"

You can add this command to your "$profile" file, so you don't need to execute this command in every newly started PowerShell window. Run the next command to add this line to your $profile automatically:
"`n" + '$env:PATHEXT += ";.py" # Transparent execution of Python scripts' |Out-File $profile -Append -Encoding Default

You can test it by creating and running a small script:

'print("Hello, World!")' | Out-File MyScript.py -Encoding Default

.\MyScript.py

This post has been moved to http://mycomputeradventures.foxinnovations.be/2012/09/05/python-scripts-inside-powershell-window/ .

Uploading a file to Amazon Glacier using PowerShell

2012-08-30T21:12:00.000+02:00

For my own convenience, I created a PowerShell module to upload backup files to Amazon's new backup service "Glacier".

The module is contained in this file:
http://dl.dropbox.com/u/2350654/blog/FxAWS.zip
(including the Amazon AWS SDK library)
Extract it into your Powershell module directory, e.g.
C:\Users\<your name>\Documents\WindowsPowerShell\Modules.
(this will create a directory FxAWS under the directory Modules)

Start powershell and run:
Import-Module FxAWS

C:\Users\wim devos.GENOFFICE> Write-AWSGlacier -AWSAccessKey '<access key>' -AWSSecretKey '<secret key>' -AWSRegion <some region> -
GlacierVault <vault name> -Filename <filename> -Description <description>

e.g.

Write-AWSGlacier -AWSAccessKey '[your access key]' -AWSSecretKey ' [your secret key] ' -AWSRegion us-east-1 -GlacierVault "backup" -Filename "backup-20120830.7z"

The parameters are:

AWSAccessKey and AWSSecretKey.
These are NOT your login and password to log on to the Amazon.com web site.
You can find the access key and secret key in your Amazon.com account:
https://portal.aws.amazon.com/gp/aws/securityCredentials
Extra credentials can be created and they can be removed individually.
AWSRegion.
The region where your Glacier Vault has been created.
GlacierVault
The name of the Vault that you created in the Glacier administration website.
FileName
The file that you want to upload.
Description (optional)
A description used in the Glacier administration website. When none is specified, the last part of the filename path is used.

You can create 3 (global) PowerShell variabled $AWSAccessKey_Default, $AWSSecretKey_Default and $AWSRegion_Default, in which case you can omit the -AWS* parameters.

GIT External Diff / Merge tools

2012-08-30T21:00:00.000+02:00

Occording to the GIT book, GIT supports these external Diff / Merge tools:

[araxis] Araxis Merge - Commercial
[bc3] Beyond Compare 3 - Commercial
[diffuse] Diffuse - Open source
[ecmerge] ECMerge - Commercial
[emerge] Emerge (Emacs) - Open source
[gvimdiff] gvimdiff - Open source
[kdiff3] KDiff3 - Open source
[meld] Meld merge - Open source
[opendiff] opendiff - OS X Developer Tools
[p4merge] P4Merge - Commercial
[tkdiff] TkDiff - Open source
[tortoisemerge] TortoiseMerge - Open source - [ Merge tool only ]
[vimdiff] vimdiff - Open source
[xxdiff] xxdiff - Open source
[kompare] Kompare - Open source - [ Diff tool only ]

You can choose your prefered diff / merge tool by executing this command (Windows or Linux), in this case for my favorite kdiff3.

git config --global diff.tool kdiff3

git config --global merge.tool kdiff3

The external diff is started using this command:

git difftool

You will probably get the error "The diff tool kdiff3 is not available as 'kdiff3'. external diff died, stopping at <filename>.", because GIT doesn't find your tool.
This can be resolved by adding your tool to the environment PATH variable, or by telling GIT where to find your tool(s):

git config --global difftool.kdiff3.path "C:/Program Files (x86)/KDiff3/kdiff3.exe"

git config --global mergetool.kdiff3.path "C:/Program Files (x86)/KDiff3/kdiff3.exe"

Any configuration can be undone by executing:

git config --global <config item> --unset

e.g.

git config --global mergetool.kdiff3.path --unset

This post has been moved to http://mycomputeradventures.foxinnovations.be/2012/08/30/git-external-diff-merge-tools_30/ .

Solving ANTLR errors using ANTLRWorks

2012-08-27T20:13:00.000+02:00

Solving ANTLR grammar errors can be very difficult, especially in complex grammar files.

Below is a simple example, based on the GQL ANTLR-grammar used in FxGqlC.
(reduced to illustrate the problem. A complete grammar can be found here. GQL is a domain language similar to SQL / T-SQL)

grammar sql;
select_command
: SELECT (WS top_clause)? WS column_list EOF
;
top_clause
: TOP expression
;
column_list
: expression (WS? ',' WS? expression)*
;
expression
: expression_3
;
expression_3
: expression_2 (WS? op_3 WS? expression_2)*
;
op_3 : '+' | '-' | '&' | '|' | '^'
;
expression_2
: expression_1 (WS? op_2 WS? expression_1)*
;
op_2 : '*' | '/' | '%'
;
expression_1
: op_1 WS? expression_1
| expression_atom
;
op_1 : '~' | '+' | '-'
;
expression_atom
: NUMBER
| '(' WS? expression WS? ')'
;
SELECT : 'select' ;
TOP : 'top' ;
NUMBER : DIGIT+;
WS
: (' '|'\t'|'\n'|'\r'|'\u000C')+
;

fragment DIGIT : '0'..'9';

The 3 expression "levels" are used to handle operator precedence. The grammar is designed to be able to parse expressions like:

SELECT 17
SELECT 17 * 14 + 3
SELECT 17 + 14 + 3
SELECT - 17
SELECT 17 * - 14 + 3
SELECT 17 + 14 + - 3
SELECT TOP 3 17
...

When trying to 'compile' or 'Interpret' the grammar in ANTLRWorks, you get this error:

[11:36:44] error(211): <notsaved>:21:43: [fatal] rule expression_3 has non-LL(*) decision due to recursive rule invocations reachable from alts 1,2. Resolve by left-factoring or using syntactic predicates or using backtrack=true option.
[11:36:44] warning(200): <notsaved>:21:43:
Decision can match input such as "WS {'+', '-'} WS NUMBER" using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input

Solving this error just by analyzing the grammar is quite a challenge, even for this very simple example. When using a large grammar file it is nearly impossible.
But ANTLRWorks has a very useful tool to show what's going wrong.

The error message indicates that there is a problem with expression_3 (expression_3 is also indicated in red in the list of rules/tokens in the left pane).
Put your cursor in expression_3, and select the tab "Syntax Diagram" in the lower pane.
First, in the lower pane, select "Alternatives '1'" in the upper right corner.
==> In green you see how the grammar matches "WS '+' WS NUMBER", which is exactly what we want.
Next, select "Alternatives '2'" in the upper right corner.
==> In red you see how the grammar matches "WS '+' WS NUMBER".
In the latter case, you can see that the matching starts in the TOP-clause.

This is what's happening: there can be an ambiguity when parsing "SELECT TOP 1 + 2 + 20".
It is not clear where the top-clause ends and the column-list starts. Both '+' signs can be unary or binary.

It can be: "SELECT [TOP 1] [+ 2 + 20]", being equivalent to "SELECT TOP 1 22"
Or it can be: "SELECT [TOP 1 + 2] [+ 20]", being equivalent to "SELECT TOP 3 20"

This ambiguity must be resolved, because only one interpretation should be valid.
In this specific case, the grammar could be changed in a way that the top-clause expression should always have parentheses surrounding it when it is not a simple number.
This can easily be achieved by changing:

top_clause
: TOP expression
;

to:

top_clause : TOP expression_atom ;

This solves the ambiquity. The text "SELECT TOP 1 + 2 + 20" is now parsed as "SELECT [TOP 1] [+ 2 + 20]".
And if somebody wants to use "1 + 2" in the TOP-clause, he should use: "SELECT TOP (1 + 2) + 20", which is parsed as: "SELECT [TOP (1 + 2)] [+ 20]"

Below you find the complete example, with the TOP-clause corrected:

grammar sql;
select_command
: SELECT (WS top_clause)? WS column_list EOF
;
top_clause
: TOP expression_atom
;
column_list
: expression (WS? ',' WS? expression)*
;

expression
: expression_3
;
expression_3
: expression_2 (WS? op_3 WS? expression_2)*
;
op_3 : '+' | '-' | '&' | '|' | '^'
;
expression_2
: expression_1 (WS? op_2 WS? expression_1)*
;
op_2 : '*' | '/' | '%'
;
expression_1
: op_1 WS? expression_1
| expression_atom
;
op_1 : '~' | '+' | '-'
;
expression_atom
: NUMBER
| '(' WS? expression WS? ')'
;
SELECT : 'select' ;
TOP : 'top' ;
NUMBER : DIGIT+;
WS
: (' '|'\t'|'\n'|'\r'|'\u000C')+
;

fragment DIGIT : '0'..'9';

FxGqlC v2.3

2012-08-21T21:37:00.000+02:00

A new version of FxGqlC has been released.

The major changes are documented here:
https://sites.google.com/site/fxgqlc/home/fxgqlc-manual/changes-in-fxgqlc-2-3

You can download FxGqlC v2.3 here:
https://sites.google.com/site/fxgqlc/home/downloads

Regular expression matching in C++11

2012-08-18T20:00:00.000+02:00

A part of the boost (http://www.boost.org/) functionality regarding regular expressions has been included in the new C++11/C++0x standard.

This code works on GCC and even on Visual C++ 10 (Visual Studio 2010) and above:
std::regex rgx("(\S+@\S+)");

std::smatch result;
std::string str = std::regex_replace(std::string("please send an email to my@mail.com for more information"), rgx, std::string("<$1>"));
// str contains the same text, but with the e-mail address enclosed between <...>.

More information on regular expressions can be found here: http://www.regular-expressions.info/ .

Replace all occurrences of a character in a std::string with another character

2012-08-17T20:00:00.000+02:00

in one line of C++ code, using C++11/C++0x:

std::string str = "my#string";
std::for_each(str.begin(), str.end(), [] (char &ch) { if (ch == '#') ch = '\\'; } );
// str now contains my\string

The for_each function code calls the lambda expression (indicated in yellow) for every character, and the lambda expression replaces the '#' with a '\'

The future of technology?

2012-08-14T22:09:00.000+02:00

Click the image to make it bigger:

Source: http://envisioningtech.com/

2012 2013 2014 2015 2016 2017 2018 2019 2020 2030 2040 2012 2013 2014 2015 2016 2017 2019 2020 2030 2040 ROBOTICS BIOTECH MATERIALS ENERGY ARTIFICIAL INTELLIGENCE SENSORS GEOENGINEERING QUANTITATIVE FORECASTS INTERNET INTERFACES UBICOMP SPACE BITS ATOMS RELATIVE IMPORTANCE CONSUMER IMPACT CLUSTER OF TECHNOLOGIES The node size indicates the predicted importance of a technology. The outline of a node indicates a consumer impact larger than the technological novelty. A jagged outline indicates a cluster of similar technologies grouped together. World population: 8 billion Source: U.N. – http://bit.ly/7nqQkS World population: 7 billion BRICs GDP overtakes the G7 Source: Goldman Sachs – http://bit.ly/nc9Wqj Petabyte storage standard Source: http://bit.ly/r9BYQc Exabyte storage standard Source: http://bit.ly/kPMKMb Terabit internet speed standard Source: http://bit.ly/kPMKMb World population: 9 billion Source: U.N. – http://bit.ly/7nqQkS Source: http://bit.ly/6MoQJc Sources: Intel – http://intel.ly/pWbH04 Ericsson – http://bit.ly/avvVok Alan Conroy – http://bit.ly/pofHp5 FutureTimeline – http://bit.ly/qz4ben Sources: Intel – http://intel.ly/pWbH04 InternetWorldStats – http://bit.ly/AKbO5 Source: U.N. – http://bit.ly/7nqQkS Global online population: ± 2 billion Connected devices: ±10 billion Global online population: 4-5 billion Connected devices: 30-50 billion $150 Hard disk: ±200 Tb Standard RAM: ±750Gb Global online population: ± 2.5 billion Connected devices: ±15 billion $ 1.000 computer reaches the capacity of the human brain (± 10 15 calculations per second) Vertical farming Weather engineering Seasteading Desalination Carbon sequestration Climate engineering Arcologies Commercial spaceﬂight Sub-orbital spaceﬂight Lunar outpost Mars mission Solar sail Space elevator Space tourism Inductive chargers Thorium reactor Traveling wave reactor Fuel cells Multi-segmented smart grids Biomechanical harvesting Bio-enhanced fuels Artiﬁcial photosynthesis Space-based solar power Piezoelectricity Photovoltaic glass Nanogenerators Enernet Tidal turbines Programmable matter Personal fabricators Molecular assembler Metamaterials Additive manufacturing Graphene Optical invisibility cloaks Biomaterials Carbon nanotubes Self-healing materials Nanowires Antiaging drugs Stem-cell treatments In-vitro meat Nanomedicine Artiﬁcial retinas Rapid personal gene sequencing Synthetic biology Personalized medicine Gene therapy Hybrid assisted limbs Smart drugs Synthetic blood Organ printing Smart toys Robotic surgery Telematics Appliance robots Self-driving vehicles Domestic robots Powered exoskeleton Embodied avatars Swarm robotics Utility fog Commercial UAVs Fabric-embedded screens Reprogrammable chips Picoprojectors Volumetric (3D) screens Flexible screens Skin-embedded screens Modular computers Tablets Boards Retinal screens Eyewear-embedded screens Context-aware computing Smart power meters Biometric sensors Machine vision Optogenetics Depth imaging Biomarkers Neuroinformatics Near-ﬁeld communication Pervasive video capture Computational photography Speech recognition Haptics 4K Augmented reality Gesture recognition Multi touch Immersive virtual reality Holography Telepresence 4G 5G Cloud computing Interplanetary internet Exocortex Photonics Virtual currencies Cyberwarfare Mesh networking Reputation economy Remote presence VR-only lifeforms Machineaugmented cognition Software agents High-frequency trading Natural language interpretation Procedural storytelling Machine translation Research & visualization by Michell Zappa mz@envisioningtech mz@envisioningtech.com mz@envisioningtech.com Envisioning emerging technology for 2012 and beyond Last updated: 2012-02-10 Understanding where technology is heading is more than guesswork. Looking at emerging trends and research, one can predict and draw conclusions about how the technological sphere is developing, and which technologies should become mainstream in the coming years. Envisioning technology is meant to facilitate these observations by taking a step back and seeing the wider context. By speculating about what lies beyond the horizon we can make better decisions of what to create today. BY SA

Export of Office Outlook contacts to GMail

2012-08-10T20:30:00.000+02:00

To import your Microsoft Office Outlook contacts to GMail or Google Apps, you need to export them first to a CSV file.

In Outlook, go to the "File" tab in the ribbon menu, and click "Options" in the left sidebar.
In the Outlook Options dialog, click on "Advanced" in the sidebar, and click the "Export" button.
In the first step of the Import and Export wizard, select "Export to a file", and click "Next".
In the second step, select "Comma Separated Values (Windows)", and click "Next".
In the third step, select your Contacts folder that you want to export (normally "Contacts"), and click "Next".
In the fourth step, enter or select the filename, e.g. "contacts.csv".
Click "Finish" to start the export.

When you import this file in GMail, and you are a member of a Windows Active Directory domain, the e-mail addresses are not imported. Instead, the e-mail address field in GMail contains the "distinguished name" of your contact as known to your ActiveDirectory. E.g. "cn=jsmith,ou=promotions,ou=marketing,dc=noam,dc=reskit,dc=com".
The real e-mail address is however included in the CSV file, as part of the column "E-mail Display Name", which contains the full name and the regular e-mail address between parentheses, but this column isn't used by the GMail import.

You could replace all E-Mail Addresses in the file using an Excel formula, or manually in a text-editor.
Or you can simply use this FxGqlC command to replace all e-mail address columns with the e-mail address taken from the display name:

select replaceregex($line, '\"/o=.*?\",\"EX\",(\".*?$(.*?)$\")', '"$2","EX",$1') into [contacts2.csv] from [contacts.csv]

The same method can be used to replace national telephone numbers into an international format:
select replaceregex($line, '\+?(32\d{8,9})', '+$1') into [meucci3.csv] from [meucci2.csv]

You need to adopt the regular expression to a format appropriate for your contacts.

Import the resulting file in GMail, and that's it.