CORSIS

Tiny F# EDSL for creating system / hardware IDs on Windows using WMI classes

2012-01-20T14:44:00.001+01:00

SystemID EDSL Module:

module SystemID =
 
    open System
    open System.Management
 
    type ClassName     = string
    type PropertyKey   = string
    type PropertyValue = obj
    type IDComponent   = ClassName *  PropertyKey                 []
    type IDQueryResult = ClassName * (PropertyKey * PropertyValue)[][]
 
    let internal qs (t:ClassName) (ps:PropertyKey[]) =
        try use mos = new ManagementObjectSearcher("Select " + String.Join(",", ps) + " From " + t)
            let col = mos.Get()
            let arr = Seq.toArray
            seq { for o in col do
                    yield seq { for p in o.Properties do yield p.Name, p.Value } |> arr } |> arr
        with _ -> [|[||]|]
    let internal gs s = s|>Array.map (Array.filter (fun e -> match e with k,null -> false |_->true))
    let internal vs (t, ns) = t,qs t ns |> gs
 
    let query : IDComponent[] -> IDQueryResult[] = fun (cs:IDComponent[]) -> cs |> Array.map vs
 
    module Conversions =
        let inline private bytes (raw:string) = System.Text.Encoding.UTF8.GetBytes raw
        let text rs =
            let b = System.Text.StringBuilder()
            rs |> Array.iter (sprintf "%A" >> b.AppendLine >> ignore)
            b.ToString()
        let sha (raw:string) =
            let sha = System.Security.Cryptography.SHA1.Create()
            raw |> bytes |> sha.ComputeHash |> System.Convert.ToBase64String
        let inline hex raw = raw |> bytes |> System.BitConverter.ToString |> fun x -> x.Replace("-", "")
 
    let inline (!/) x = "Win32_" + x

Create custom system ID based-on operating system serial number:

> open SystemID

  open SystemID.Conversions;;

> [| !/"OperatingSystem", [| "SerialNumber" |] |] |> query;;

val it : IDQueryResult [] =
  [|("Win32_OperatingSystem",
     [|[|("SerialNumber", "00477-179-7015276-84376")|]|])|]

> [| !/"OperatingSystem", [| "SerialNumber" |] |] |> query |> text |> sha |> hex;;

val it : string = "433855487A6D4F76453845736667777061724A6E5336734F4434513D"








Create custom hardware ID based-on processor and BIOS information:

>   [|

        !/"Processor", [| "Architecture"; "Caption"     ; "Name" |]
        !/"BIOS",      [| "ReleaseDate" ; "SerialNumber"         |]
    |] |> query;;

val it : IDQueryResult [] =
  [|("Win32_Processor",
     [|[|("Architecture", 9us);
         ("Caption", "Intel64 Family 6 Model 30 Stepping 5");
         ("Name", "Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz")|]|]);
    ("Win32_BIOS", [|[|("ReleaseDate", "20090731000000.000000+000")|]|])|]

>   [| 
        !/"Processor", [| "Architecture"; "Caption"     ; "Name" |]
        !/"BIOS",      [| "ReleaseDate" ; "SerialNumber"         |]
    |] |> query |> text |> sha |> hex;;

val it : string = "544E50614E4A3579784168496C6E4445683843684D6A72314F41493D"







Choose and create system / hardware IDs exactly as you wish!

Corsis Research

2011-11-05T23:59:00.001+01:00

2011 - Logo

Solidity, growth, and new dimensions!

WCF 4.0 routing with MetadataExchange using System.ServiceModel.Routing.RoutingService in F#

2011-03-07T16:11:00.001+01:00

I posted a simple question on stackoverflow.com:

I have service with working MEX at:

net.tcp://remotehost:4508

What is the shortest C#/F# code (hard time understanding XML configuration files ^_^") I could write to create a router to it at?:

net.tcp://localhost:4508

MEX should also be routed properly so that clients can use the router

svcutil net.tcp://localhost:4508

to discover service methods.

and as has been usual with its users no one has been able to answer my question before I could come up with my own answer.

Here's how to use F# and WCF 4.0 to route a service with NetTcp and MexTcp endpoints no XML involved, pure code:

namespace CORSISmodule Application =    open System    open System.ServiceModel    open System.ServiceModel.Routing    open System.ServiceModel.Dispatcher    open System.ServiceModel.Description    let createSimpleRouter createBinding (routerAddress : string) serviceAddress =         let routerType = typeof<IRequestReplyRouter>        let routerContract = ContractDescription.GetContract(routerType)        let endpoint address = new ServiceEndpoint(routerContract, createBinding(), new EndpointAddress(address))        let serviceEndpoints = [| endpoint serviceAddress |]        let configuration = new RoutingConfiguration()        configuration.FilterTable.Add(new MatchAllMessageFilter(), serviceEndpoints)        let host = new ServiceHost(typeof<RoutingService>)        ignore <| host.AddServiceEndpoint(routerType, createBinding(), routerAddress)        host.Description.Behaviors.Add(new RoutingBehavior(configuration))        host            [<EntryPoint>]    let main(args) =        let (routerAddress, serviceAddress) =            match args with            | [| ra; sa |] -> (ra, sa)            | _ -> ("net.tcp://localhost:4508/", "net.tcp://remotehost:4508/")        let netTcp() = new NetTcpBinding(SecurityMode.None)        let mexTcp() = MetadataExchangeBindings.CreateMexTcpBinding()        let tcpRouter = createSimpleRouter netTcp  routerAddress           serviceAddress        let mexRouter = createSimpleRouter mexTcp (routerAddress + "mex") (serviceAddress + "mex")        tcpRouter.Open()        mexRouter.Open()        Console.WriteLine("routing ...\n{0} <-> R:{1}", serviceAddress, routerAddress)        ignore <| Console.ReadKey true        0

Posted via email from corsis

Words that are Hard to Say - Reverse Video Search with Mathematica and TinEye

2010-12-12T00:12:00.003+01:00

It all started with this link: http://www.facebook.com/video/video.php?v=150827548298158. My brother wrote to me saying he had seen this movie trailer and would like to watch it but the only link he had was of this facebook video which had no names mentioned.

I had a simple idea:

extract a given number of frame images
use TinEye to look for pages using similar images

I used Mathematica to extract the frames.

Then, using TinEye, we performed a reverse image search on certain frames we picked manually.

http://www.tineye.com/search/5728ec915156cc8f0568d27d5109bd9b9c79cbf9/
http://www.tineye.com/search/6a97500eef0bc33220555e02a8bd1f5500157a6e/ (here we actually got the result!! but overlooked it)
http://www.tineye.com/search/b2ecfa195ca5bc2883dcc63e53e2b307d3b739a2/
http://www.tineye.com/search/02b352b008c9ff144695745a630e0c3a30d492e3/
http://www.tineye.com/search/0f748c782fc992ddd8ac5cd8b5bfec323a933048/

http://www.tineye.com/search/18c255ad6a07e6a1a3f128d2ef942393a47ae3cb/ (here we did a combined search using the first 4 result-returning images from above, and finally succeeded to see the name we were looking for)

It was a fun experience! In the end we learned it was not a movie but a music video from a Korean artist.

And here's the music video:

Mathematica-TinEye-VideoSearch

Posted via email from CORSIS

Oh Dear Photoshop CS5: "Content-Aware Fill Me that Big ** Whale!"

2010-10-28T22:28:00.001+02:00

Having some fun with my new CS 5 ... it's beautiful. Bring on the next Christmas party good god! We have got kids to turn into pirates and ninja and what not ...

Posted via email from CORSIS

XML Key/Value Cache Optimizations

2010-09-02T03:18:00.001+02:00

For a read-only XML file server with a simple path-based addressing scheme, I initially used ConcurrentDictionary<string, XElement> as a key/value cache. This generated and kept constantly in memory a lot of unnecessary XAttribute and XElement instances which contributed nothing to raw data transfers across the wire. Considering the fact that resident XAttribute or XElement instances in the file server process were totally useless because we do indexing and serve result paths via our query server, I switched to using path-indexed string buffers with path-to-segment dictionaries as the new key/value cache.

Memory footprint for hundred documents, before/after

See and download the full gallery on posterous

My question on Stackoverflow.com with some code

http://stackoverflow.com/questions/3613713/how-to-get-stream-positions-of-xml-elements-in-net

Full working environment screenshots ^__^

See and download the full gallery on posterous

Posted via email from CORSIS

ELIT Private Alpha

2010-08-16T02:12:00.003+02:00

Launching ELIT Client for private alpha testing of political and social scientists from Heidelberg and Mannheim.

Posted via email from CORSIS

OCR Comparison - FineReader 10 vs Acrobat 9

2010-07-17T02:33:00.001+02:00

A silent midnight comparison of two OCR software.

See and download the full gallery on posterous

Posted via email from CORSIS

Hash Tables in Mathematica

2010-04-30T06:50:00.000+02:00

This week, after downloading citing and related paper numbers of around 50.000 papers that were presented as results to various simple searches in PubMed, we wanted to have a way to find out most cited authors.

Both extremely new to Mathematica, we were yet to see anything that looked or behaved like hash tables so our first approach looked like this:


recordsOfAuthor[author_]:=
  Select[
    records,
    Composition[MemberQ[#, author]&, authorsOf]
  ]

where records were a list of XMLElements containing EndNote X3 records (these roughly correspond to PubMed search results).

Now I am not the guy to immediately understand the O-complexity of any given code, yet the first thing I had told my friend after we had finished downloading the citation numbers and right before we started coding the rest of our analysis tools was that we definitely needed to either find out how to use hash tables in Mathematica or do the whole thing in another language, otherwise we might have some headache.

Our largest EndNote X3 search results file was a single 95MB XML document with 17.278 records (papers/reviews/etc.) and 52.666 unique authors, combined with another 8MB XML document containing the citation and related paper counts we had downloaded from PubMed for each record.

x-axis: record number, y-axis: number of citations

In another part of the code we have a map over all unique authors that uses the above function to sum the citation numbers of all records for every single author.

Let's consider the worst case complexity now:

52.666 unique authors
17.278 records
79.159 authors

The latter two makes each call of recordsOfAuthor cost 17.278 x 79.159 = 1.368.500.792 (1 billion, 368 million, 500 thousand and 792) operations! So with our 52.666 unique authors, we are talking about a whopping: 52.666 x 17.278 x 79.159 = 72.031.772.832.532 operations to find out all records of each author!!!!!!!!! 72 trillion, 31 billion, 772 million, 832 thousand and 532 operations!!!!!!!!!!!

Our recordsOfAuthor was O(n^3). No surprise that it made me wait for the results a whole night ... whose sweet slumber brought upon me a vision of myself being scolded and harshly criticized by my friend for being the slowest and most unsuccessful coder she has ever seen, then googling for a way to use hash tables in Mathematica and seeing myself go to stackoverflow and reading an answer.

Now let's have a look at this:


Clear[dic, dicAdd];
dic[accNum_] := Null (* dictionary *)
dicAdd[record_] := dic[accNumOf @ record] = record;

dicAdd /@ records; (* 17.278 operations *)

Clear[ar, arAdd, recordsOfAuthor];
ar[author_] := {} (* ar: author records *)
arAdd[record_] := (ar[#] = Append[ar[#], accNumOf @ record)& /@ authorsOf @ record;

arAdd /@ records; (* 79.159 operations *)

recordsOfAuthor[author_] := author // ar // dic /@ # & (* O(1) *)

Now we first do 17.278 + 79.159 = 96.447 operations to create an O(1) version of recordsOfAuthor by using function definitions as a hash table.

With this version of recordsOfAuthor the same map over all unique authors has become O(1) * n = O(n), so we have reduced the number of operations

from 72 trillion, 31 billion, 772 million, 832 thousand and 532
to just 52 thousand, 666!!!!!!

With all the being hated, blamed and made felt worthless by my best friend and one week of delay in development time, this has been an excellent lesson: about experience, understanding, patience, indifference, simplicity and trust – I will never forget her words: "You always say, you'll finish, you'll finish but you never do!!"!

PageRanking in Mathematica

2010-04-22T19:06:00.011+02:00

In the "Advanced Topics in Information Retrieval" course I'm taking this term, we had a light introduction to PageRank algorithm and I wanted to redo the relevant slides and solve some exercises in Mathematica - which I happened to learn while providing programming support to my best friend in her endeavor of mathematical modeling and simulation of drug resistance in infectious diseases!.

Here it is as PDF and Mathematica Notebook!

It's truly amazing what you can achieve with Mathematica with tiny amounts of code. I find the selection of functions provided in version 7 and their implementations to be stunning examples of programming done right.

2009 - 2010

2009-12-27T11:38:00.003+01:00

No news ... I asked someone to not tell me any so I will not post any personal news either. I'll be in Turkey for a week, meeting new friends and spending 2009's last and 2010's first days in their company.

CORSIS .2 First Applications

2009-10-26T10:58:00.005+01:00

As part of a research project I am involved in, the development version of CORSIS is now being used and actively extended with latest technologies. For more information, visit our new wiki.

erlang ! hello

2009-02-11T12:49:00.020+01:00

On Tuesday while reading Programming Erlang on suggestion of Andreas Cardeneo from the Research Center for Information Technology of the University of Karlsruhe, I started to experiment in Haskell with typed channels and lightweight threads to imitate Erlang style processes and networks thereof.

Here is the very early and raw code from a few hours of exploration.

And here is a simple interactive session in which a server is created that reads strings into integers. These integers then get _distributed one at a time_/_dealt_ to a first layer of 3 parallel nodes and they then travel to a second layer:

*Erlang> (server,out) ← serve (read :: String → Integer)
*Erlang> l1ts ← create 3 :: IO ([Chan Integer])
*Erlang> did  ← deal move ([(\y → y - x) | x ← [1..3]]) out l1ts
*Erlang> l2ts ← create 3 :: IO ([Chan Integer])
*Erlang> tids ← sequence $ zipWith (link move (2 *)) l1ts l2ts
*Erlang> server ·· [ show x | x ← [2..4] ]
*Erlang> all_ flush l2ts
-- [[2],[2],[2]]
{-
   FLOW VIEW

            l1    l2                     
            ---   ---
             1  →  2
in    out   ---   ---
---   --- ↗    
"2"    2    ---   ---
"3" →  3  →  1  →  2
"4"    4    ---   ---
---   --- ↘    
            ---   ---
             1  →  2 
            ---   ---

-- before "all_ flush l2ts" actually only l2 is not empty

   ACTUAL STATE

            l1    l2                     
            ---   ---
                →  2
in    out   ---   ---
---   --- ↗    
            ---   ---
    →     →     →  2
            ---   ---
---   --- ↘    
            ---   ---
                →  2 
            ---   ---
-}

one could use such a system to distribute, say, processing of images to multiple parallel threads that utilize different cores or processors on a single machine. Using copy links (link copy ...) instead of move links (link move ...) would leave copies of the intermediate results on channels and we could see into the individual stages of a complex image manipulation. Tagging messages that travel between such processes would make it possible to divide a problem into pieces and process the pieces in parallel and put the results back together. One could make the inter-process communication lazy and declare a process network structure that produces results only when there is demand from the terminal nodes.

Check out this link for more examples of different process networks!

As I write these lines, I don't know if there is any prior Haskell work about such things but I could imagine this or a more serious endeavour turning into a nice little framework for distributed processing. By using more advanced primitives for communication than forkIO and Chan a, one might span a process network across machine boundaries.

In this blog post, 'process' means "a lightweight thread that reads from a channel and applies a function on what it has read and writes the result to another channel" and NOT "OS process".

This protoprototype of process networks has not been performance-tested in any way. This is very much an experiment and further investigation of the concept may or may not turn out to be useful.

erlang ! hello

2009-02-10T07:11:00.003+01:00

Here is the clock example from Programming Erlang (page 155) in Erlang [clock.erl] and its very rudimentary approximations!? in Haskell [clock2.hs, general2.hs].

Mono 2.0 binary package for Solaris 10/x86

2009-02-06T21:29:00.003+01:00

Just received some fresh Mono 2.0 binaries for Solaris 10/x86. Many thanks pablo! Mono 2.2 to follow after the weekend *^o^*!

Debugging Haskell: LFG, AVM → DAG

2008-07-24T16:54:00.008+02:00

A few days ago I had to do some debugging in Haskell while writing a function to draw directed acyclic graphs of attribute value matrices from LFG F-structures. UnsafePerformIO was great help there. I did most of this during a machine translation lecture given by Kurt Eberle of lingenio.

Here are the debugging functions:

deb1 :: Show a ⇒ a → a
deb1 a = unsafePerformIO (print a >> return a)

deb2 :: Show a ⇒ String → a → a
deb2 msg a = unsafePerformIO (putStrLn (((msg ++) . show) a) >> return a)

deb3 :: (Show a, Show b) ⇒ (a → Bool) → b → a → a
deb3 p b a = unsafePerformIO $ esc p b a
  where
    esc p b a | p a       = putStr ((show b) ++ (show a)) >> return a
              | otherwise = return a

cond = (== "^OBL")
dobj = deb3 cond

And here is how I used them in context:

core :: Integer → String → [(Integer,String)] → AVM → (String,[(Integer,String)],Integer)
core i s ls (M att (A val)) = (s ++ x i ++ " [label=\"" ++ dobj s att ++ "\"];\n",(i,val):ls,i)
core i s ls (M att (C mms)) = let (xs,xls,xi) = add i i s ls att mms in 
  (s ++ x i ++ " [label=\"" ++ att ++ "\"];\n" ++ xs,xls,xi)
    where
      add f i s ls att []     = ([],[],i)
      add f i s ls att (m:ms) = (cs ++ rs,cls ++ rls,ri)
        where
          (cs,cls,ci) = core    ni sm ls m
          (rs,rls,ri) = add  f  ci [] ls att ms
          sm = x f ++ " -> "
          ni = i + 1

I know core pretty much sucks in many ways o_O!! Still trying to figure out how I could keep such things maintainable in Haskell... (I will have a look at the FGL sometime soon.)

It was very easy to use type classes and unicode infix functions to achieve a really pretty syntax to represent F-structures though ^o^:

s1 :: AVM
s1 =  void ≈
     [pred ≈ "antworten<(^SUBJ)(^OBL)>",

      subj ≈ [pred ≈ pro,
              num  ≈ sg, 
              cas  ≈ nom ],

      obl  ≈ [pred ≈ "auf<^OBJ>",

              obj  ≈ [pred ≈ "Brief",
                      num  ≈ sg,
                      cas  ≈ acc]    ],

      tense ≈ present                   ]

Machine translation is the last lecture before sommer holidays *^o^*! I will have more time to work on Corsis again.

mono string hash code collisions

2008-07-17T09:17:00.007+02:00

After reading the following blog post by David R. MacIver about hash code collisions in java strings:

For reasons you either know about by now or don’t care about, I was curious as to how well String’s hashCode was distributed (I suspected the answer was “not very”). I ran a few quick experiments to verify this.

For your amusement, here is a list of all hash collisions between alphanumeric strings of size 2: http://www.drmaciver.com/collisions.txt and here is a list of all which don’t collide with any others http://www.drmaciver.com/noncolliding.txt

Some statistics: There are 3844 alphanumeric strings of size 2. Of these 3570 collide with at least one other string. That is, 274 of these strings (or about 7% of them) *don’t* collide with something else.

Oh well. It’s a good thing no one would be stupid enough to rely on hashCode to distinguish the contents of two objects.

I tested things with .NET 3.5 and MONO 1.9.1 on a 32-bit Windows Vista:

running on .NET 3.5:
3844 two-char strings
14776336 comparisons
0 collisions with one or more items
0 total collisions

running on mono 1.9.1:
3844 two-char strings
14776336 comparisons
**3570** collisions with one or more items
5250 total collisions

2-char-long-string-hashcode-collisions-mono-1.9.1

We then both went on to testing hash code collisions in 3-char-long strings. David in java:

For what it’s worth, even fewer strings have unique hash codes for 3 characters. 3948 don’t collide, or about 1.6% of them.

Also, this of course doesn’t mean that probability of a hash collision is really high. In reality it’s acceptable low. It’s just a demonstration that it’s not hard to find colliding pairs.

and I in mono 1.9.1: 3-char-long-string-hashcode-collisions-mono-1.9.1. 3948 don't collide and the colliding pairs seem to be the same ones again.

Microsoft's implementation seems to be doing a much better work here. I will inform mono devs soon about the situation ^_^

Mono 1.9 binary package for Solaris 10/x86

2008-04-01T17:16:00.005+02:00

Jonel Rienton has provided a Mono 1.9.0 binary package for Solaris 10/x86! I am thankful to him for his kindness^_^!

okitsune 狐

2008-02-21T06:43:00.018+01:00

I am working on a new project called: okitsune 狐 - it is going to be a simple theorem prover of sorts for logic students. Okitsune is being written in Haskell and open for contributions: okitsune.sourceforge.net

code sample: analytic tableaux for propositional logic

Haskell and F#: Language Design

2008-01-05T03:45:00.001+01:00

After watching a taste of Haskell by Simon Peyton-Jones and having a look at A History of Haskell: being lazy with class, I am left with some open questions regarding the design of F#:

Why is F# not lazy?: What is the advantage of eagerness?

Is there a point in encapsulating side-effectful operations in monads while programming in F#? IO Monads etc.?

#light

// Sample F# IO Monad : OSCON Haskell Video Part 2

type IO<'a> = IO of 'a

//  get: unit -> IO<string>
let get () = IO(read_line    ())

//  put: string -> IO<unit>
let put x  = IO(print_string x)

//     bind.e: IO<'a>  ->  'a
// bind: IO<'a> -> ('a -> IO<'b>) -> IO<'b>
let bind (x:IO<'a>) (y:('a -> IO<'b>)) = let e (IO(a)) = a in y (e x)
let (>>=) = bind

//  gp: unit -> IO<unit>
let gp () = get () >>= put

gp ()

Why did the F# team chose ML and OCaml over Haskell?

I hope someone from the F# team enlightens me on all these. I am still scrutinizing the 55-pages-long history of Haskell to create a list of questions I want to ask.

Segmenter Compiler: Benihime, 紅姫

2008-01-03T00:45:00.001+01:00

Code generation subsystem of Benihime is undergoing major refactoring with two goals: extensive use of generics in the public API and maximum amount of code reuse.

With major concern about the performance of the current regex implementation in Mono (1.2.6), I wish I already had enough time to spare today to submit a standards-compliant regex compiler replacement as a contribution. I hope Benihime to become one in the near future.

Expert F#

2008-01-03T00:44:00.000+01:00

On a completely unrelated note, I have been reading Expert F# and I must admit that I begin to like the language. If you are someone who is going to learn his first programming language or just feel like adding a new one to your arsenal, I strongly recommend having a look at F# – with interesting mix of paradigms, terse syntax and strong type inference, it is sure to be a pure delight to work with.

Mono 1.2.6 binaries for Solaris 10/x86

2008-01-01T18:18:00.000+01:00

Duncan Mac Leod from Tucan Entertainment has been kind enough to share his Mono 1.2.6 binaries for Solaris 10/x86.

Levenshtein Distance Algorithm: Fastest Implementation in C#

2007-12-02T13:04:00.000+01:00

Here is a cleaned-up performance test for several different implementations of levenshtein I have blogged about recently. This test was emailed to me by Ahmed Ghoneim, who has also kindly agreed to its publication on my blog. I am very grateful to him for his excellent contribution. I have slightly altered his file to do away with the unnecessary local variables in my C2C# port of the GNULevenshtein method.

I would like to hear from you which methods perform best on your machine. Please drop a comment ^_^!

LevenshteinAlgorithmPerformanceTest.cs
code only

Packages
code, data and sample binary in zip and self-executable zip formats

Please note that the GNULevenshtein method was found to be buggy! Here is the new replacement method.

Perplexity in Markov N-Gram Models

2007-11-26T00:55:00.001+01:00

While implementing the perplexity function on Markov n-gram models as it is described on page 14 of Jurafsky & Martin's SLP to appear, I came across some floating point overflow, underflow issues and had to come up with equations to avoid them.

Here is my solution in detail and its bigram implementation.

It took me a lot of whiteboarding and a few hours to figure this one out but the resulting libcorsis code is %100 foreign intellectual property free ^_^.

I am now looking for ways to analyze distributions graphically and testing different encapsulations of probability values and their interaction with the public API.