defgeneric

Clojure: different JVM options for different JVMs

2019-10-12T10:12:00.000+02:00

I sometimes use OpenJ9 instead of Hotspot -- and so I need different JVM options depending on what VM / compiler I use. Here's one way of doing this; in ~/.lein/profiles.clj

{:user
{:global-vars {*warn-on-reflection* true, *unchecked-math* :warn-on-boxed, *print-length* 100}

:plugins [[lein-ancient "LATEST"]]

:jvm-opts
#=(eval
(concat
["--illegal-access=deny"
"-XX:-OmitStackTraceInFastThrow" ;; See http://stackoverflow.com/a/2070568/160305 ..and this is also interesting http://blogs.atlassian.com/2011/05/if_you_use_exceptions_for_path_control_dont_fill_in_the_stac/
]

;; OpenJ9 stuff
(when (clojure.string/includes? (System/getProperty "java.vm.name") "OpenJ9")
["-Xquickstart"
;; TODO: https://github.com/eclipse/openj9/issues/4205
(str "-Xshareclasses:name=" (last (clojure.string/split (System/getenv "PWD") #"/")))
"-Xscmx1g" ;; https://www.eclipse.org/openj9/docs/xscmx/

])))}}

OpenWRT 18.06.1 on the TL-MR3020

2018-10-19T22:45:00.002+02:00

The new OpenWRT images are too big for the old TL-MR3020, but by removing IPv6 support I got these working just fine on my (HW ver. 1.9): https://drive.google.com/drive/folders/1f6XLDQLhvdm9SyTQeFWjZ8tf9f6QCtpR?usp=sharing

..I can provide other custom builds of OpenWRT if people are interested.

....some notes to self:

make -j4 image PROFILE="tl-mr3020-v1" PACKAGES="-ip6tables -odhcp6c -kmod-ipv6 -kmod-ip6tables -odhcpd-ipv6only -luci-proto-ipv6 kmod-usb-core kmod-usb2 -kmod-usb-ohci luci"

..using the `tiny` images.

Extract audio or music as is (copy) from video using FFmpeg

2014-04-25T20:01:00.004+02:00

I keep forgetting how to do this. Note to self:

ffmpeg -i input.mp4 -vn -c:a copy output.aac

Also, here's how I make sure music downloaded from e.g. YouTube can be played on my phone:

for f in *.m4a; do ffmpeg -y -i "$f" -acodec copy -movflags faststart "$f.aac"; done

Xubuntu 14.04: Black screen (Light Locker)

2014-04-23T15:02:00.001+02:00

Yeah, Xubuntu 14.04 shouldn't have shipped with Light Locker. They should have used XScreenSaver for now. Here's how to fix black screen after unlocking or resuming after suspend:

sudo apt-get install xscreensaver

sudo apt-get remove light-locker light-locker-settings

There's an issue talking about this here: https://bugs.launchpad.net/ubuntu/+source/xfce4-power-manager/+bug/1259339 ..but none of the suggestions there worked for me.

The Xubuntu devs will probably fix this later, and I'll switch to Light Locker then; when it actually works; not before.

Clojure: How to detect transaction (STM) retries

2014-04-07T11:03:00.001+02:00

Here's a simple snippet that gives you an idea how to detect transaction (STM) retries in Clojure:

(let [n (atom 0)]
(dosync
(if-not (zero? @n)
(println "try >= 2 of transaction")
(do
(swap! n inc)
(println "first try of transaction")))))

...of course this won't trigger retries, but you get the idea.

Essential tools for telecommuting

2013-10-08T21:43:00.000+02:00

..yet another work in progress post as of Sep 29. 2013.. ..will update this later!

Software

First of all; Mosh: http://mosh.mit.edu/

Mosh improves work via SSH drastically. I run Emacs inside Tmux on servers; highly recommended. Anyone who works while on the road must try Mosh. Each connection dropout (e.g. while on a train) will seem invisible to you!

Also http://justgetflux.com/ will probably keep my eyes in working condition for another 10+ years. UPDATE: http://jonls.dk/redshift/ is better than flux!

Wi-Fi amplifier and external antenna

I'm currently using:

Amplifier: Alfa AWUSO36NH
Antenna: TP-LINK TL-ANT2408C

..I'm considering a directional antenna too. Heck this setup might be a good idea in any case as I sit with my laptop on my, well, lap a lot. I.e., I can disable the internal Wi-Fi radio to avoid radiation.

To get the most out of a setup like this, I do this to increase radio power and disable Wi-Fi power saving etc.:

sudo iw reg set BO
sudo iwconfig wlan1 power off
sudo iwconfig wlan1 txpower fixed
sudo iwconfig wlan1 txpower 2000mW

Laptop

Avoid glossy screens.
Encrypt your home partition!
Don't use a swap partition – or if you do (get more memory instead); encrypt this too.
Investing in a SSD instead of HDD is very much worth the extra money needed. If the laptop I like comes with a HDD I remove it and put it in an external cabinet (USB).

Mobile phone dedicated to role as Wi-Fi router (tether)

Avoid iPhones as these cannot tether in all cases.
Get a decent Android phone; make sure it has a strong 2G/3G/4G radio.
You'll need two phones if you need to be available on your normal phone number.

Communication: text

Where privacy is not important I use Gmail.
Where privacy does matter I use Pidgin or Jitsi with encryption (OTR): http://en.wikipedia.org/wiki/Off-the-Record_Messaging

Communication: voice

I'm currently using Mumble. You can setup your own server on a cheap VPS. Make sure you're using the latest version with support for the Opus codec as this will save you bandwidth: https://launchpad.net/~mumble/+archive/release
You'll probably need a headset – as echo cancellation does not always work too well!
Adjust your Mumble settings; I use the lowest quality with full noise cancellation:

quality: 8.0 kb/s
audio per packet: 60ms
noise suppression: -60dB
max amplification: 1.00

I find Jitsi or perhaps Pidgin interesting as I'm not 100% happy with Mumble for various reasons.

..that's pretty much it – I think. Stuff like an issue tracker, Git etc. should be used by everyone regardless of whether they are telecommuters or not.

Ubuntu 13.04 (Raring Ringtal) and Intel GPU; fix for slow graphics performance

2013-04-30T06:32:00.002+02:00

I just did an upgrade to 13.04 and noticed that the graphics performance was sluggish. Creating an /etc/X11/xorg.conf file and putting this in it then rebooting helped here (ThinkPad L520):

Section "Device"
Identifier "Card0"
Driver "intel"
Option "AccelMethod" "uxa"
#Option "AccelMethod" "sna"
EndSection

Update: I also had problems mounting USB disks etc.. Here's how I solved that on my machine:

sudo ppa-purge ppa:langdalepl/gvfs-mtp

SSH: Forwarding, etc.

2013-04-05T19:06:00.002+02:00

I'm going to write down how to do port forwarding and other things with SSH here – because I keep forgetting how to do it, and wasting 10 minutes reading man-pages and Googling around every time I do is getting old. I'll add to this post in the future.

Basic port forwarding

All connections on port 9998 on localhost (127.0.0.1) will now go to port 5432 on 100tb.nostdal.org:

ssh -fNT -L 9998:127.0.0.1:5432 fod@100tb.nostdal.org

This'll create a background process – and if your connection goes stale for whatever reason killing it works. Look it up using ps -AF | grep ssh

AutoSSH

AutoSSH can be used to make sure a tunnel, or set of tunnels in this case, stays up:

/usr/bin/autossh -fNT -L 4444:127.0.0.1:4444 -L 4445:127.0.0.1:4445 -L 7657:127.0.0.1:7657 -L 6668:127.0.0.1:6668 -L 7654:127.0.0.1:7654 -R 4242:127.0.0.1:80 afoss.nostdal.org

The above will setup several tunnels. 4445 locally will be forwarded to 4445 on aafoss.nostdal.org and so on. I've put this string in "Application Autostart" under "Sessions and Startup" in Xubuntu (XFCE) so my tunnels are up after a reboot also.

Opus 1.1-x (git) audio test

2013-02-16T13:07:00.000+01:00

Source

Source:
http://www.lindberg.no/hires/test/2L48SACD_14_stereo_96k.flac (85.8MB)
from http://www.2l.no/hires/index.html

Encoded using Opus from Git (02.16.2013) with standard options (i.e. VBR):

$ time opusenc 2L48SACD_14_stereo_96k.flac 2L48SACD_14_stereo_96k.opus
Encoding using libopus 1.1-alpha-28-g6044907 (audio)
-----------------------------------------------------
Input: 96kHz 2 channels
Output: 2 channels (2 coupled)
20ms packets, 96kbit/sec VBR
Preskip: 352

Encoding complete
-----------------------------------------------------
Encoded: 5 minutes and 13.82 seconds
Runtime: 8 seconds
(39.23x realtime)
Wrote: 4291832 bytes, 15691 packets, 316 pages
Bitrate: 108.504kbit/s (without overhead)
Instant rates: 85.2kbit/s to 194kbit/s
(213 to 485 bytes per packet)
Overhead: 0.827% (container+metadata)

real 0m8.631s
user 0m8.545s
sys 0m0.064s

Result

The resulting Opus file is 4.3MB, here:
http://100tb.nostdal.org/nostdal.org/audio/2L48SACD_14_stereo_96k.opus

The same result encoded in FLAC for convenience since Opus cannot be played back everywhere, yet:
http://100tb.nostdal.org/nostdal.org/audio/2L48SACD_14_stereo_96k-opus.flac

...can you tell the difference between the source and the result? http://en.wikipedia.org/wiki/ABX_test

Logitech C920 webcam: H264 full HD mode stream to file on Linux

2013-01-25T01:53:00.003+01:00

This works using https://launchpad.net/~videolan/+archive/master-daily :

vlc 'v4l2:///dev/video1:chroma=H264:width=1920:height=1080:fps=30'

To stream this to a file:

cvlc 'v4l2:///dev/video1:chroma=H264:width=1920:height=1080:fps=30' --sout '#file{dst=webcam-test.mp4,no-overwrite}'

Databases as an extension of memory: STM + DB transactions = ♥

2013-01-10T05:45:00.001+01:00

..or how databases doesn't really need to be regarded as I/O

The problem with I/O is that it's tricky when combined with transactions. Since memory transactions (STM) need to retry on conflicts just like database transactions do -- one end up with multiple I/O calls:

user> (let [r (ref nil)]
(future
(dosync
(ref-set r "one")
(println "one")
(Thread/sleep 1000)))
(dosync
(ref-set r "two")
(println "two")
(Thread/sleep 1000)))
two
one
two
nil
user>

The same problem applies when dealing with a database from an outside transaction too; one'll end up with multiple entries for e.g. the same person and so on.

..but there's a way around this. Databases have something "normal" I/O doesn't have; they have their own transactions. We can run two transactions at the same time in parallel; one for data in memory and one for data in the database -- then commit them together, or indeed roll back both if either detects a conflict.

This is tricky to get right, and for this to actually work both transactions need support for intermediate pre-commit stages where things won't roll back. Only when both transactions are in these stages at the same time can full finalization of the commits be done. If either fails because of a conflict or anything; both roll back.

In PostgreSQL, this stage can be entered via prepared transactions. On the Clojure end, things are a bit trickier as Clojure doesn't have anything like this per say -- but Ref validators will work! Check out the do-mtx function here:
https://github.com/lnostdal/SymbolicWeb/blob/4f3304721b32b3db1e87d4a6d20c2318ec499a7b/src/symbolicweb/database_common.clj#L114

..further, the process that'll make these two transactions dance together via these stages is known as a two-phase commit. This is done in the do-2pctx function here:
https://github.com/lnostdal/SymbolicWeb/blob/4f3304721b32b3db1e87d4a6d20c2318ec499a7b/src/symbolicweb/database_common.clj#L137

Rolling back the transactions are done using exceptions, and the newer Clojure constructs ex-info and ex-data are used to communicate the intent of the roll-back; abort or retry. I must admit I sometimes miss the Common Lisp condition and restart system here, but this will do just fine..

Anyway, this of course opens up for some interesting performance related problems since the Clojure STM is optimistic, but the sheer convenience and simplicity of this is worth it in some cases. Later, when or if needed, I'm considering adding a pessimistic STM to Clojure to improve this. I have a very crude and quite horribly written one laying around for Common Lisp: https://github.com/lnostdal/SW-STM, but I think it has some core potential.

..OK, that's it for now. I might post some examples to show this stuff off or whatever later.

Clojure STM: ON-COMMIT construct

2012-12-22T17:08:00.000+01:00

I've sometimes missed an ON-COMMIT like construct for the Clojure STM.

Agents work with the Clojure STM, but sometimes using Futures seem more fitting as an Agent only executes one action at any point in time. Also, exceptions in Agents won't roll back Clojure side transactions..

The problem with using Futures directly from a DOSYNC context, of course, is that they will be executed or "spawned" several times as the STM does its retries.

I'm not sure this is the best or only way to do something like this, but anyway:

https://www.refheap.com/paste/7814/fullscreen

(def ^:dynamic *on-commit-fns*)

(defmacro on-commit [& body]
`(swap! *on-commit-fns* conj (fn [] ~@body)))

(defmacro with-on-commit-ctx [& body]
"It is possible to do I/O in the ON-COMMIT bodies. If an exception is thrown in an ON-COMMIT the MTX (DOSYNC) will roll back."
`(binding [*on-commit-fns* (atom [])]
(let [on-commit-fns# (ref nil)]

(set-validator! on-commit-fns#
(fn [v#]
(when v#
(try
@(future
(doseq [^Fn on-commit-fn# v#]
(on-commit-fn#)))
(catch Throwable e#
(throw (clojure.stacktrace/root-cause e#)))))
true))

(let [res# (do ~@body)]
(ref-set on-commit-fns# @*on-commit-fns*)
res#))))

;;; Testing it:

user> (let [r (ref 0)]
(future
(dosync
(with-on-commit-ctx
(println "frst: trying..")
(ref-set r 1)
(let [r-value (ensure r)]
(on-commit (println "first:" r-value)))
(Thread/sleep 500))))

(future
(dosync
(with-on-commit-ctx
(println "second: trying..")
(ref-set r 2)
(let [r-value (ensure r)]
(on-commit (println "second:" r-value)))
(Thread/sleep 1000)))))
#<core$future_call$reify__3390@1bfb3209: :pending>

frst: trying..
second: trying..

second: trying..
second: trying..
second: trying..
second: trying..
first: 1
second: trying..
second: 2

user>

Hello Haskell World

2012-12-04T21:31:00.000+01:00

I sometimes wonder why I don't play around with Haskell more. Perhaps I'd miss the Lisp syntax -- and the macros? The run-time seems very interesting from what little I know about it. Perhaps it would be "too pure" for my tastes? Rust seems interesting too ...

$ ls
hello.hs

$ cat hello.hs
module Main where

main = putStrLn "Hello, World!"

$ ghc --make -O2 hello.hs -o hello
[1 of 1] Compiling Main ( hello.hs, hello.o )
Linking hello ...

$ ls -l
total 1036
-rwxr-xr-x 1 lnostdal lnostdal 1048016 Dec 4 21:20 hello
rw-r--r- 1 lnostdal lnostdal 1218 Dec 4 21:20 hello.hi
rw-r--r- 1 lnostdal lnostdal 51 Dec 3 04:54 hello.hs
rw-r--r- 1 lnostdal lnostdal 3552 Dec 4 21:20 hello.o

$ ./hello
Hello, World!

$ strip -s hello
$ ls -l
total 736
-rwxr-xr-x 1 lnostdal lnostdal 737312 Dec 4 21:20 hello
rw-r--r- 1 lnostdal lnostdal 1218 Dec 4 21:20 hello.hi
rw-r--r- 1 lnostdal lnostdal 51 Dec 3 04:54 hello.hs
rw-r--r- 1 lnostdal lnostdal 3552 Dec 4 21:20 hello.o

$ ldd hello
linux-vdso.so.1 => (0x00007fff6b1ce000)
libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f8ded458000)
libffi.so.5 => /usr/lib/x86_64-linux-gnu/libffi.so.5 (0x00007f8ded24b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8decfc8000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8decdc0000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8decbbc000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8dec831000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8dec615000)
/lib64/ld-linux-x86-64.so.2 (0x00007f8ded6dd000)

$ ./hello
Hello, World!

$

Skype 4.1 on Debian "testing" / "wheezy" on AMD64 arch

2012-12-01T13:49:00.001+01:00

Microsoft has their cold hands on Skype now so how long it will be around no one knows. For now, they still release Linux and Android versions -- and this works with the latest (as of this post) 4.1 version using Debian "testing" / "wheezy" on AMD64:

dpkg --add-architecture i386
apt-get update
apt-get install libpulse0:i386
wget -O skype-install.deb http://www.skype.com/go/getskype-linux-deb
dpkg -i skype-install.deb
apt-get install -f

Linux screencasting: FFmpeg, H264, VP9, Opus

2012-10-03T11:55:00.000+02:00

I'll collect various and hopefully useful ways of doing screencasts or desktop recordings on Linux here. The focus is not on screencasts for e.g. gaming here; for this you should up the FPS among other things.

Note that you'll need to adjust the -s option to match the screen resolution of your setup. Also note that I'm assuming 10 FPS is OK; this can adjusted with the -r option.

Lossless H264 (video) + Opus (audio) => Matroska (MKV) container

The -preset option can be set to medium if your CPU can handle it (for smaller file size), but something like this:

ffmpeg -thread_queue_size 1024 -f alsa -i pulse -f x11grab -r 10 -s 1920x1080 -i :0.0 -c:a libopus -c:v libx264 -qp 0 -preset ultrafast desktop-recording-$(date -u +%Y-%m-%d-%s).mkv

When done you can and should compress it further:

nice ffmpeg -i desktop-recording.mkv -c:a copy -c:v libx264 -qp 0 -preset veryslow desktop-recording-final.mkv

The -qp parameter can be adjusted if you want to go from lossless (0) to lossy (> 0) compression; higher values means lower quality.

By prefixing with the nice command, the ffmpeg program will run more as a background task not disrupting other work you want to do while this finishes.

Lossless VP9 (video) + Opus (audio) => Matroska (MKV) container

The new VP9 codec takes up too much CPU on my now old laptop, so I stick with lossless H264 (above) when recording – then reencode the resulting file to VP9 after I'm done recording:

nice ffmpeg -i desktop-recording.mkv -c:a copy -c:v libvpx-vp9 -lossless 1 desktop-recording-final.mkv

Well compressed VP9

The above is lossless encoding. For screencasting it seems doing very strong recompression like this is generally "good enough" and leads to very small file sizes. This will also transcode your audio from stereo to mono (-ac 1) in Opus format:

nice ffmpeg -i desktop-recording.mkv -c:a libopus -ac 1 -b:a 64k -c:v libvpx-vp9 -b:v 0 -crf 50 -tile-columns 4 -threads 8 desktop-recording-final.mkv

Smaller values for crf means better quality.

VLC H264 desktop and audio recording or streaming via HTTP on Linux

2012-10-02T16:48:00.001+02:00

Update: Also see this post for an example using FFmpeg; VLC has been giving me a hard time with this in some cases.

HTTP stream

cvlc screen:// --screen-fps=10 --input-slave=pulse:// --sout "#transcode{venc=x264,vcodec=h264,fps=10,vb=600,acodec=mp3,channels=2,ab=128,audio-sync}:std{access=http,mux=ts,dst=/stream.mp4}"

Now you can watch: vlc http://localhost:8080/stream.mp4

Use pavucontrol to select input device.

Stream to file

cvlc screen:// --screen-fps=10 --input-slave=pulse:// --sout "#transcode{venc=x264,vcodec=h264,fps=10,vb=600,acodec=mp3,channels=2,ab=128,audio-sync}:std{access=file,mux=mp4,dst=desktop-cap-$(date -u +%Y-%m-%d-%s).mp4}"

This stuff is sort of a black art; working examples are hard to come by. I hope it's useful.

http_build_query from PHP to Clojure

2012-04-27T23:17:00.001+02:00

This is based on http_build_query from PHP ( http://php.net/manual/en/function.http-build-query.php) and basically builds URL-encoded query strings from (nested) Clojure maps and vectors.

There's probably an already existing lib with a function for this around on Github somewhere, but its friday and I wrote this for fun...

Anyway, here's the code: https://refheap.com/paste/2421/fullscreen

Reloading or refreshing a web page really fast; a "third refresh method"

2011-12-08T09:13:00.001+01:00

Perhaps some or even most of you out there know this, but there seems to be at least three ways of reloading or refreshing a page in Chrome / Chromium -- and in Firefox too as far as I can tell. The two commonly known ones are (I'm using Linux here; Windows and Mac might have different key bindings):

Ctrl-r, which does a normal reload; polling the servers for updates (with help from If-Modified-Since or etags or similar) for each external resource, and loads from local browser cache on HTTP 304 Not Modified type responses.
Ctrl-Shift-r, which does what's commonly known as a hard refresh; fully reloading all content and external resources, completely ignoring the browser cache.

But there's another way of refreshing a page that's even faster than Ctrl-r :

Press Alt-d then press Enter -- or press Ctrl-l then press Enter.

This sort of does what Ctrl-r does, except it doesn't check ("poll") whether the servers has any updates with regards to external resources. It'll do a normal request for the resource directly referred to in the address bar of course -- which means most dynamic data will be fetched anew, but external resources are to a much larger extent fetched from the local cache.

JavaScript
This refresh type can be done via JavaScript also. Instead of doing window.location.reload(); do window.location.href = window.location.href; but note that this will not work if your URL has a fragment identifier (hash part).

PostgreSQL: ALTER TABLE ==> "cannot be cast"

2011-12-07T18:31:00.001+01:00

Altering a column type in PostgreSQL involves some cast trickery. As a note to myself and perhaps others Googling around for solutions, here goes:

ALTER TABLE "my_table" ALTER COLUMN "product_price" TYPE bigint USING 0::bigint;

My preferred software stack

2011-08-26T03:41:00.004+02:00

Last update: 9 September 2012

I get questions about my preferred software stack from time to time, so here it is -- and for further reference:

Linux (Debian, Xubuntu, Android): On server, desktop, laptop and phone. Xubuntu is really the only sensible thing to use on the desktop or laptop at the moment; both for beginners and "advanced" users.
Clojure (on the JVM): Well thought out and designed.
Netty: Scalable I/O for the JVM.
lighttpd: Excellent as a reverse-proxy for dynamic back-ends as it is the only front-end that does proper I/O multiplexing with its back-ends last time I checked.
PostgreSQL: http://www.wikivs.com/wiki/MySQL_vs_PostgreSQL also see http://www.youtube.com/watch?v=1PoFIohBSM4
..and I target the Web: HTTP, HTML(5), CSS, JavaScript (ECMAScript), AJAX, Comet (and WebSocket) +++.

I do know and use other technologies, but I prefer to interop with them or use them only when strictly needed, e.g. C for some embedded platforms -- as opposed to doing my main development in or with them.

Here's some more:

Emacs for all editing etc., with nREPL and nREPL.el as a Clojure front-end.
tmux
GCC for C, with GDB, Kdbg, DDD and Valgrind for debugging.
Git: distributed (source code) revision control system.
rsync, and the front-end Grsync. I use these for both backup purposes and for keeping my desktop and laptop in sync.

Others (Twitter!) moving to the JVM also

2011-07-26T13:09:00.004+02:00

It seems I'm not alone having struggled with issues related to Garbage Collection. Others, even Twitter, are also moving to the JVM: http://www.youtube.com/watch?v=ohHdZXnsNi8&feature=player_detailpage#t=219s

Also see https://dev.twitter.com/blog/twitter-open-source-and-jvm

Update: some more links;

Twitter (Ruby -> Scala) - http://www.readwriteweb.com/hack/2011/07/twitter-java-scala.php
Foursquare (PHP -> Scala) - http://www.scala-lang.org/node/5130
Facebook (PHP/C++ -> PHP on JVM) - http://nerds-central.blogspot.com.au/2012/08/facebook-moving-to-jvm.html
Tumblr (PHP -> Scala) - http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html
Square (Ruby -> Java/JRuby) - http://www.quora.com/Square-Inc-1/What-is-the-technology-stack-behind-Square

Also check out the other OSCON 2011 videos rolling in right now:
http://www.youtube.com/user/OreillyMedia#g/c/93FC98105B19725C

Update: In particular loving the Steve Yegge talk.

..and so it begins; SymbolicWeb has started its migration to Clojure

2011-07-19T23:56:00.010+02:00

This'll just be a quick announcement since the state of this thing doesn't warrant anything else at the moment, but I've begun the transition of SymbolicWeb to Clojure. The Git repository can be found here: https://github.com/lnostdal/SymbolicWeb

For some background on why I'm doing this, see the previous posts here and here.

Even though this thing is very incomplete, any comments or e-mails regarding the code even at this point would be most welcome.

...the following is probably more or less a set of notes to self:

I'm not happy with how some of things are "bootstrapped". The FIND-OR-CREATE-APPLICATION-INSTANCE and similar stuff seems off.
The HTTP request and response need to be decoupled. This'll lead to way better design and performance. A look at HANDLE-COMET-REQUEST should reveal the reason for this.
I'm not sure how to go about the above yet. I'm thinking I should in the process probably switch to a better back-end right away; something like Netty, but then again Real Life issues are pressing on so maybe later.
Obviously I need some sort of URL-based routing so the same JVM instance can host multiple applications.
The low-level stuff for dealing nicely with HTML and CSS doesn't exist yet.
There are no widgets here at all.
No MVC system or anything exists yet.

..that's it for now; good night!

Ubuntu: how to deal with (or not) Unity

2011-07-02T20:08:00.011+02:00

UPDATE: Since this was written I've moved more and more to XFCE: http://www.psychocats.net/ubuntu/purexubuntu

Problem

The recent changes in Ubuntu have been problematic for me and others. I'm talking of course about the major switch from Gnome 2.x to Unity. It's a complete paradigm shift in that they are trying to unite how one interact with software on the desktop and on a typical netbook, tablet/pad or phone. Unfortunately it doesn't work very well.

It feels sort of embarrassing to have recommended this OS to other people as something that is simple, stable and "just works and stays that way" when Ubuntu goes about this major change. Also, running "Classic Mode" seems silly as Gnome 2.x probably won't be supported for long.

What to do?

edit: I've switched to XFCE (Xubuntu) since I wrote this post.
Well, I decided to switch to KDE or Kubuntu. Yup. ...and even though I've never used KDE before it has been a very positive experience actually.

I didn't actually do a complete reinstall of the OS, opting instead to simply remove all Gnome related packages and replace them with KDE ones. I followed this excellent guide:

http://www.psychocats.net/ubuntu/purekde

Anyway, multiple monitor support works as it should; I can place panels on each monitor and only windows from each separate monitor shows up in the task manager widgets placed on their respective panels.

Alt+F2 pops up a simple "Search/run program" dialog at the top of the screen which is fast and has way more features than what was found in the vanilla Gnome 2.x "Run program" dialog.

For video and movie playback I prefer SMPlayer instead of the standard KDE Dragon Player. SMPlayer is of course available in the package manager. Just hit Alt+F2 and start typing "package" and KPackageKit will show up; hit enter.

Another benefit is that I don't end up with all the bloatware Canonical adds to the typical standard Gnome desktop for Ubuntu.

A sort of rolling distro

Another great thing about KDE or Kubuntu is that they have a PPA for the desktop packages:

sudo apt-add-repository ppa:kubuntu-ppa
sudo apt-add-repository ppa:kubuntu-ppa/backports
sudo aptitude update
sudo aptitude dist-upgrade

This means I get the best of many worlds. I get a stable set of core packages from Ubuntu with updates with regards to security related issues and bug fixes related to typical driver issues -- and, I get the latest and greatest stable desktop release.

Also, installing Google Chrome will automatically configure the repository settings necessary to keep itself up-to-date. Which again is great; both desktop and browser is up-to-date, while the more core packages stays rock solid and stable.

Updating core packages is what makes a 100% rolling release distro more complex to deal with anyway; I don't need or want that.

Clojure as a way out

2011-05-08T15:40:00.005+02:00

As briefly mentioned in the previous post I don't think Clojure is very mature, but it still seems to be the best way out for me. In a way the JVM really is a or the killer feature. The JVM run-time, the API it provides and the available external libraries and projects is just too good and complete to disregard when combined -- even with Clojure itself still being sort of an infant.

I tried CCL for a while, but it lacks a lot of non-trivial stuff like timers, support for atomic operations and more; stuff that's really needed to do what I want to do. It also seems to have some "weird bugs" ಠ_ಠ of its own which I won't bring up here. In the end I can't bring myself to trust it, and the community around it seems stale compared to SBCL and Clojure.

Some negative notes on Clojure:

I don't see why it doesn't have support for classic OO style programming in addition to what it already supports seeing as it has a STM.
I'm not sure I agree with how the STM is implemented. It has been a long time since I last looked at Clojures STM though; perhaps things have changed. I recall thinking it could cause resource starvation for instance -- and what about write skew? I'm not sure I agree with the solution to this, but perhaps the main idea with Clojure anyway is to use its STM only as a sort of fallback and instead mainly focus on the other concurrency support inherent in the language. Still not sure how realistic this is ...
The Slime integration isn't really there yet.
..there's probably more; I'll add more points later as I encounter them..

On a positive note it seems possible to generate code of at least decent quality with regards to performance now: http://www.skuro.tk/2011/01/24/performance-boost-in-clojure-1-3/.

So anyway, I've decided to spend the summer with the new, now in April, Clojure book: The Joy of Clojure: Thinking the Clojure Way. I'm really hoping this will work out as I'm at quite a loss as to what to do next if this fails. I have thought about and dismissed, some fully others close, many other potential ways out and ideas already.

Here's the Clojure version of the code posted at the end of the previous post:

user> (let [obj (java.lang.ref.WeakReference. (list 42))]
        (System/gc)
        (.get obj))
nil
user>

(..there might be some Clojure wrappers for doing this built into Clojure, clojure-contrib or some other libraries..)

On being stuck

2011-04-25T23:53:00.003+02:00

I'm stuck -- really stuck. You see, SBCL has a conservative GC which leads me down dead ends constantly while I'm trying to find the cause of a leak. As a sort of note to self:

You do not want to end up with memory related problems when using a language or language implementation with a conservative GC.

..and of course it seems memory or resource related problems are just bound to pop up from time to time when dealing with bigger projects, so being able to debug this properly is important.

This has taught me a very painful lesson for sure. I haven't written a line of code in months.

I'm not sure what I should do next; perhaps port everything to CCL? I've also been thinking about Scala, but non-Lisp doesn't seem very "fun", and I cannot seem to take Clojure seriously. I don't know ...

Using trivial-garbage:

CL-USER> (lisp-implementation-type)
"Clozure Common Lisp"
CL-USER> (let ((x (tg:make-weak-pointer (list 42))))
           (tg:gc :full t)
           (tg:weak-pointer-value x))
NIL
CL-USER>

CL-USER> (lisp-implementation-type)
"SBCL"
CL-USER> (let ((x (tg:make-weak-pointer (list 42))))
           (tg:gc :full t)
           (tg:weak-pointer-value x))
(42)
CL-USER>