English | Gamlor

Korean Hangul Stopped Working After IBus Update to 1.5.18? A Work Around.

gamlerhart — Sun, 06 May 2018 00:47:26 +0000

When I’m using Linux I use IBus to write Korean. A few days ago when I updated my Manjaro installation writing Korean stopped working. After booting everything looked fine. The IBus tray icon was there, switching languages with Win+Space seemed to work.

OK!

Win+Space=OK

However, when I started to type the IBus icon suddenly changed and it wrote in English. I didn’t know that icon and it seemed strange. There was no error message. IBus just always switched to that icon and then wrote in the English alphabet.

What is this?

Well, first I searched the Internet without success. I found only old entries which did no help. So, maybe there are logs? I couldn’t find any IBus log in /var/logs or something like that. However, the IBus man page told about a –verbose flag. So, I used that and tada, there were error messages:

[gamlor@gamlor-t470p ~]$ ibus-daemon --verbose -r &
[4] 13270
[gamlor@gamlor-t470p ~]$ 
(ibus-daemon:13170): GLib-WARNING **: 21:33:08.140: GChildWatchSource: Exit status of a child process was requested but ECHILD was received by waitpid(). See the documentation of g_child_watch_source_new() for possible causes.
(ibus-engine-hangul:13296): IBUS-WARNING **: 21:33:08.224: org.freedesktop.IBus.Config.GetValue: GDBus.Error:org.freedesktop.DBus.Error.Failed: Config value [engine/hangul:on-keys] does not exist.
(ibus-engine-hangul:13296): IBUS-WARNING **: 21:33:08.224: org.freedesktop.IBus.Config.GetValue: GDBus.Error:org.freedesktop.DBus.Error.Failed: Config value [engine/hangul:off-keys] does not exist.
(ibus-daemon:13270): GLib-CRITICAL **: 21:33:08.229: g_variant_unref: assertion 'value != NULL' failed
(ibus-ui-gtk3:13277): IBUS-WARNING **: 21:33:13.143: ibus_bus_call_sync: org.freedesktop.DBus.Properties.Get: GDBus.Error:org.freedesktop.DBus.Error.Failed: No global engine.

Oh…’assert failed’ doesn’t sound good. And whenever I tried to write Korean, the `bus_bus_call_sync…No global engine` message got printed. So I searched for that error message and found this Github issue. There is a work around: When configuring an Emoji selection key, the issue goes away. Strange. So, I did that and everything worked again. Here’s how: IBus Preference -> Emoji -> … Button -> Select a impossible key combination -> Add -> Done!

Work around

The end

The post Korean Hangul Stopped Working After IBus Update to 1.5.18? A Work Around. first appeared on Gamlor.

Async Database Connector for Java (ADBCJ) on JCenter.

gamlerhart — Tue, 24 Oct 2017 11:55:57 +0000

Due to lack of time and interest, ADBCJ is in ‘maintenance’ mode. However, I still improve it from time to time =).

ADBCJ will be fine ^.^

ADBCJ in JCenter

Finally, ADBCJ is in a decent Maven repository! That is a long standing to do crossed of the list. First, you’ll need to add JCenter to your projects maven repos. Unless your build tool has the JCenter by default.
For Maven:


 
 false
 
 central
 bintray
 https://jcenter.bintray.com

After than you include ADBCJ as a dependency.

MySQL driver:


    org.adbcj
    adbcj-api
    0.9


    org.adbcj
    mysql-async-driver
    0.9

H2 driver:


    org.adbcj
    adbcj-api
    0.9


    org.adbcj
    h2-async-driver
    0.9

Changes

Java 8

Recent ADBCJ versions requires Java 8. Instead of ADBCJ’s own future implementations, methods now return Java 8’s completable futures =).

Callback API

Java 8’s futures are not every bodies taste. Especially other JVM language have their own futures and concurrency constructs. Or you might really do not want a future based control flow. So, I changed ADBCJ’s basic API to be callback driven. That allows to use other concurrency constructs without going through Java futures as intermediate step. The Java 8 completable future methods are also implemented in term of these futures.

Examples on Github

Examples how to use ADBCJ are on GitHub.

The post Async Database Connector for Java (ADBCJ) on JCenter. first appeared on Gamlor.

Need a CDN? Try out edgemesh.com.

gamlerhart — Mon, 09 Oct 2017 14:01:24 +0000

Do you need a CDN for your website? Maybe you want to try edgemesh.com. The basic subscription is free!

Start

First sign up on edgemesh.com. Then embed this script into your website:

Done. Really!

So, how does Edgemesh work?

A normal CDN has tons of servers all over the world. Those server caches your images, CSS and Javascript. So, visitors can download your content quickly from a close CDN server. However, if there is no CDN server in your visitors area, they won`t benefit from quick downloads. And CDN servers are not free!

Classic CDN

Edgemesh works differently. Nowadays our phones and computers are really fast. Our web browsers are powerful. Browsers have an IndexeDB database and can do P2P networking with WebRTC. This is what Edgemesh uses. When visitors visit a website, Edgemesh stores some content in their browser. When another nearby visitor needs the same content, they download it from other visitor`s browsers. Therefore, if you have many visitors, they can fetch the content quickly from each other.

Edgemesh

However, we can not trust the visitor`s internet connection. Maybe it is very slow, maybe it is suddenly interrupted. So, Edgemesh prefetches some content and falls back to regular downloads if it is to slow. Also, you can add Edgemesh ‘supernodes’, which act a bit like a regular CDN servers. Furthermore, you can combine Edgemesh with a traditional CDN.

Does it work?

Yes. This blog has close to no visitors, yet Edgemesh manages to store and distribute some content:

This sites Edgemesh network

Just try it out on your site. =)

The post Need a CDN? Try out edgemesh.com. first appeared on Gamlor.

Adding Timestamp to Logs

gamlerhart — Wed, 27 Sep 2017 09:24:46 +0000

How many small script or process do you have which just log to stdout/stderr? A few, dozens?
Their running as cronjobs or service?
Stashing the logs somewhere, like:

./produces-some-log.sh 2>&1 > log-of-my-script.log

Do you have time stamps on those logs?
If not, it is trivial to add. You just have to pipe the log through a time stamp tool.

Timestamp your lines

Linux: ts

On most Linux distros there the `ts` tool. If it is not installed, it is usually in the ‘moreutils’ package. So get it via `apt-get`, `yum`, `pacman` or whatever you`re distro uses.

Then, pipe the output through `ts`:

[gamlor@gamlor-manjaro log-util]$ ./produces-some-log.sh 2>&1                                          
We are making progress. We are at step 1/5         
And some other info about the system:  13:39:24 up 2:10, 2 users, load average: 1.44, 1.74, 1.76       
We are making progress. We are at step 2/5         
And some other info about the system:  13:39:25 up 2:10, 2 users, load average: 1.44, 1.74, 1.76       
We are making progress. We are at step 3/5         
....

Will become:

[gamlor@gamlor-manjaro log-util]$ ./produces-some-log.sh 2>&1 | ts                                     
Sep 25 13:39:30 We are making progress. We are at step 1/5                                             
Sep 25 13:39:30 And some other info about the system:  13:39:30 up 2:10, 2 users, load average: 1.65, 1.78, 1.77                                                                                               
Sep 25 13:39:31 We are making progress. We are at step 2/5                                             
Sep 25 13:39:31 And some other info about the system:  13:39:31 up 2:10, 2 users, load average: 1.65, 1.78, 1.77                                                                                               
Sep 25 13:39:31 We are making progress. We are at step 3/5                                             
...

You can specify a date format string to have more clear format:

[gamlor@gamlor-manjaro log-util]$ ./produces-some-log.sh 2>&1 | ts '%Y-%m-%dT%T%z'
2017-09-25T22:47:25+0900 We are making progress. We are at step 1/5
2017-09-25T22:47:25+0900 And some other info about the system:  22:47:25 up 2:18, 2 users, load average: 1.25, 1.29, 1.53
2017-09-25T22:47:26+0900 We are making progress. We are at step 2/5
2017-09-25T22:47:26+0900 And some other info about the system:  22:47:26 up 2:18, 2 users, load average: 1.25, 1.29, 1.53
...

Last, I recommend using the UTC time zone, to avoid confusion. Expecially when your servers are all over the world. Or you need to cross reference time stamps with other logs.
If your server isn`t set to UTC, you can set the `TZ` enviroment variable:

[gamlor@gamlor-manjaro log-util]$ export TZ=UTC
[gamlor@gamlor-manjaro log-util]$ ./produces-some-log.sh 2>&1 | ts '%Y-%m-%dT%T%z'
2017-09-25T13:49:54+0000 We are making progress. We are at step 1/5
2017-09-25T13:49:54+0000 And some other info about the system:  13:49:54 up 2:21, 2 users, load average: 0.66, 1.13, 1.44
2017-09-25T13:49:55+0000 We are making progress. We are at step 2/5
...

For more details, check the man pages.

For all Unix-like systems, AWK to the rescue!

BSD, illumos or some other Unix like system? Maybe there no `ts` util around? Good old awk has got you covered. On each line, print the date, then the rest of the line.

This method is more portable. AWK is installed most systems. So, if you plan do distribute your script, this is the way to go.

[gamlor@gamlor-manjaro log-util]$ ./produces-some-log.sh 2>&1 | awk '{ print strftime("%Y-%m-%dT%T%z"), $0;  }'                                                                                                     
2017-09-25T22:57:54+0900 We are making progress. We are at step 1/5                                    
2017-09-25T22:57:54+0900 And some other info about the system:  22:57:54 up 2:29, 2 users, load average: 1.29, 1.23, 1.39                                                                                      
2017-09-25T22:57:55+0900 We are making progress. We are at step 2/5                                    
...

Again, I recommend using UTC. Your server probably already uses UTC anyway:

[gamlor@gamlor-manjaro log-util]$ export TZ=UTC    
[gamlor@gamlor-manjaro log-util]$ ./produces-some-log.sh 2>&1 | awk '{ print strftime("%Y-%m-%dT%T%z"), $0;  }'                                                                                                     
2017-09-25T13:59:23+0000 We are making progress. We are at step 1/5                                    
2017-09-25T13:59:23+0000 And some other info about the system:  13:59:23 up 2:30, 2 users, load average: 1.09, 1.15, 1.35                                                                                      
2017-09-25T13:59:24+0000 We are making progress. We are at step 2/5                                    
...

That`s it =).
Thanks for the Stackoverflow answers, which include both solutions.

The post Adding Timestamp to Logs first appeared on Gamlor.

Java’s Enum.values() Hidden Allocations

gamlerhart — Thu, 31 Aug 2017 13:47:29 +0000

I recently profiled an app of mine. During profiling, I’ve noticed tons of array allocations. Something like:

Well easy, a quick grep for TypeInfo[] array. Nothing. Ah, quick grep for TypeInfo collections. Nothing! Huh? Where the hell do I allocate the array then?
Ok, more profiling! So, I enabled flight recorder with ‘-XX:+UnlockCommercialFeatures -XX:+FlightRecorder’. Then did a recording with Java Mission Control, capturing the allocation stack traces:

Ok, the TypeInfo.values() method allocates the memory. TypeInfo is a enum, and I’m using the built in values() method. The code looked something like this:

package info.gamlor.blog; import java.util.ArrayList; import java.util.List; import java.util.Random; enum TypeInfo { UNKNOWN(0), INT(1), STRING(2); private final int id; TypeInfo(int id) { this.id = id; } public static TypeInfo findType(int type){ for (TypeInfo t : values()){ if(t.id == type){ return t; } } return UNKNOWN; } } public class Main { private static Random rnd = new Random(); public static void main(String[] args) throws Exception { List parsedData = new ArrayList<>(); for (int i = 0; i < 1000000000; i++) { int idParsedFromWire = readTypeIdFromData(rnd); TypeInfo type = TypeInfo.findType(idParsedFromWire); // Pretend we use the type for the exampe parsedData.add(type); consumeData(parsedData); } System.out.println(parsedData.size()); } private static void consumeData(List parsedData) { // Parsed data is consumed if(rnd.nextBoolean()){ parsedData.clear(); } } private static int readTypeIdFromData(Random rnd) throws InterruptedException { // Data comes from the wire....let's run this example a bit slower int idParsedFromWire = rnd.nextInt(16); Thread.yield(); return idParsedFromWire; } }

Do you see the allocation? Why should it allocate that many arrays? The Enum values are constant, so it should be able to initialize an array once?
Ok, let’s take a closer look and decompile the class file:

javap -c TypeInfo.class | grep -A 10 values

And we get:

  public static info.gamlor.blog.TypeInfo[] values();
    Code:
       0: getstatic     #1                  // Field $VALUES:[Linfo/gamlor/blog/TypeInfo;
       3: invokevirtual #2                  // Method "[Linfo/gamlor/blog/TypeInfo;".clone:()Ljava/lang/Object;
       6: checkcast     #3                  // class "[Linfo/gamlor/blog/TypeInfo;"
       9: areturn

As we can see, it deferences a static array of the Enum values. Then it clones that array before returning it, causing an array allocation. Why does it do that? It is a defensive copy:
Java’s arrays are a mutable. If values() just returns the reference to the original array, then some code might change the array’s content. That creates havoc, since it would change what Enum.values() returns. Therefore, the defensive copy.

Safety First, Copy It

So, after seeing this, it is easy to fix our code. We call Enum.values() once, cache the array and use it for the parsing:

// Enum.values() does a defensive copy and allocates an array on each call // To avoid that allocation, we call values() only once private static final TypeInfo[] ALL_VALUES = values(); public static TypeInfo findType(int type){ for (TypeInfo t : ALL_VALUES){ if(t.id == type){ return t; } } return UNKNOWN; }

Et voila, the allocations are gone =)

The post Java’s Enum.values() Hidden Allocations first appeared on Gamlor.

Deploying JVM in tiny containers, be carefull!

gamlerhart — Tue, 25 Apr 2017 14:59:39 +0000

I`ve deployed my rohrli.gamlor.info service into the smallest possible triton container. Except me, noone is going to use it anyway ^-^. This container has 128Mbyte of memory, and 1/16th CPU time. The OpenJDK JVM is known for consuming tons of memory. And it is somewhat true. However, the JVM also runs in a small container. In this blog post I give some recommendations. The problems exists in small Docker and Triton containers. I show everything for both.

Chrome & JVM eat memory generously

Web App Example

The web app is just a ‘Hello World’ app. For better explaination, it also consumes tons of memory. I created the example in Scala, but the other JVM languages have the same issue. The source code for the example is here.

We compile and package the program into a Docker container. The Docker file is simple:

The we create the image with `build` and run it with `run`. With regular Docker we restrict the memory with the ‘-m 128m’ option to 128Mbyte. And with ‘–cpus 0.065’ we only give very little CPU time.
When using Triton, the ‘-m 128m’ option will find a sufficient large compute instance. Alternativly, you also can use ‘–label com.joyent.package=g4-highcpu-128M’ to create a tiny instance.
Once we started out container, we create some load with the ‘watch’ and ‘curl’ commands. Then, let`s watch the logs.

First Issue: Program Dies

The Triton instance won`t expirience any issue. But the plain Docker instance (Linux 4.10, Docker 17.04) suddenly stops working. =( The program got killed.

Hmmm….Why did it got killed? Aha…the memory is small, so we might ran out of memory and got killed. We can, while the container runs, inspect the memory usage of the JVM with `jstat`:

The penguin and whale are dangerous. Be careful!

jstat is a bit confusing. All numbers are in KByte. The JVM has a few memory pools and each memory pool has a minimum (**MN), maximum (**MX) and current size (**C).
So, inside the regular Docker container it create really big memory pools. On my computer (16GB, Linux 4.10, Docker 17.04), OGCMX (old generation heap) memory is about 2.5GByte. However, on the Triton instance it is about 40Mybte large. So, with a 2.5GByte pool, if it actually uses more than 128MByte, it will die. For now, Docker just kills processes which use too much memory. However, the JVM sees all the hosts memory, so it creates giant memory pools.
However, the Triton container lies more convincingly, so the JVM sees the containers memory sizes and creates a sufficient small memory pool.
Well, there a solution to this issue. We just configure a small memory size with the `-Xmx` option. Plus I configure the JVM such that when it runs out of memory, it dumps the memory and dies.

FROM openjdk:8u121-jdk-alpine # Assume already compiled for now COPY target/scala-2.12/useless-demo-service.jar /app/useless-demo-service.jar # Defaults for a tiny container ENV MEMORY="64m" EXPOSE 8080 CMD java -Xmx$MEMORY -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError="kill -9 %p" \ -jar /app/useless-demo-service.jar

After doing this, the JVM runs in the Docker container without any issue.

So Many Threads

Now, we configured the memory properly. However, there maybe more things we can configure properly? So, let`s see what threads are runing in the container with `jstack`:

Triton isn`t a fun guy!

Howly! More than 30 threads are running. On my computer the regular Docker container has 8 `GC task thread`s, and 4 `CompilerThread`s. On Triton there are 15! The JVM saw a lot of CPU cores and created many threads to take advantage of it. However, our container is restricted to 0.065 cores. Those many threads won`t help. Of course, we can configure this:

FROM openjdk:8u121-jdk-alpine # Assume already compiled for now COPY target/scala-2.12/useless-demo-service.jar /app/useless-demo-service.jar # Defaults for a tiny container ENV MEMORY="64m" \ CPUS=1 EXPOSE 8080 CMD java -Xmx$MEMORY -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError="kill -9 %p" \ -XX:CICompilerCount="$(($CPUS>2?$CPUS:2))" -XX:+UseSerialGC \ -jar /app/useless-demo-service.jar

In this really small container the `ParallelGC` is not really needed and we can use the `SerialGC`. In case you use a bit larger container, you can use `-XX:-XX:+ParallelGCThreads=$CPU` for `ParallelGC`. Or `-XX:G1ConcRefinementThreads=$CPU` in case you use `-XX:+UseG1GC`.
And we can use `-XX:CICompilerCount=$CPU` to reduce the compiler threads. However, it needs to be at least 2.

Still Many Threads

We still have many ‘acceptor’ and ‘selector’ threads. On Triton we’ve 4 acceptor and 4 selector threads:

Those threads are created by the web server, so you have to configure the web server you are using. In this example Jetty is used, and we configure it via simple Java properties:

And also change the Docker file:

And now, it starts to look fine, right?

Be Still Carefull of Many Threads

Our example web app now runs fine. But keep an eye on the thread count. Other Java/programming language library might create a thread pool and many threads. However, you can configure these libraries too.

Update 21th June, 2017

Well, first the OpenJDK team is improving the situation. As of JDK 8u131, the JVM does limit the CPU’s in Docker. And with the experimental flags ‘-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap’ it does limit the memory.

Also, Matt Rasband has a good blog post on how to estimate the memory usage of the JVM and set the flags properly.

Have fun running the JVM in a tiny container =)

The post Deploying JVM in tiny containers, be carefull! first appeared on Gamlor.

Transfer Files via Web: curl https://rohrli.gamlor.info

gamlerhart — Sun, 16 Apr 2017 12:53:51 +0000

Did you ever have this situation: You’re SSH’ed into a box. Or worse, you’re on some corporate *hmmkkmm slow* VPN, over which you SSH’ed into a box. Or worse, corporate VPN, use RDP to some Windows server, on that opening some tool thing which has a console to a virtual machine? And you need to want to transfer some 10’s of megs or more debug data out?
If you’re lucky, you get a slow transfer going. If you’ve bad luck, there no sane way to directly transfer the data.
Then you waste time trying to use a ‘send files via web’ service…oh god. And in the end you might move data via some server you control =(.

Why can’t I just user CURL?

Yes, why can’t I just use use CURL. Like, I ‘upload’ the data via curl, and download it at the other end. Like a simple pipe. No signing up, sending emails which links to the ‘large file’, no need for a browser.

I basically just wanted something like.

gamlor@machine-a: cat my-data | curl
gamlor@machine-b: curl > my-data

Nothing more. It does need to store it. Just a file transfer!

File Transfer Trouble

curl https://rohrli.gamlor.info

So I built it for myself =). And here it is: Röhrli, pipe files via web: .

You start by pointing CURL at it. Do not miss the S in httpS =). It will then give you a guide. A example:

curl https://rohrli.gamlor.info

>    Röhrli: Pipe your files via Web.
> ⁼
>    cURL guide:
>
>    Basics:
>    - Shows this help text. (or web page in a browser)
>        curl https://rohrli.gamlor.info
>    - Pipe a file to röhrli. It will respond with an URL:
>        cat my-file | curl -N --data-binary @- https://rohrli.gamlor.info
>    - Gzip if needed:
>        cat my-file | gzip | curl -N --data-binary @- https://rohrli.gamlor.info
>    - The request will give you an URL. Then read from it on another place:
>        curl {the-url-returned} > my-file
>
>    Any question? Email: roman.stoffel@gamlor.info. Twitter: @gamlerhart

Then you pipe your file / data to curl. As told by the instruction. After that, you’re receive instructions how do download the data.
Do not stop CURL. Röhrli does not store the file on the server. It just transfers data from the uploader to the downloader.

echo data > test-file.txt
cat test-file.txt | curl -N --data-binary @- https://rohrli.gamlor.info

>    You're pipe was created at https://rohrli.gamlor.info/vykkev-xizras-puzwyq-vauveq
>
>    Download it with:
>    - curl https://rohrli.gamlor.info/vykkev-xizras-puzwyq-vauveq > my-file.txt
>    If you uploaded gzipped, decompress:
>    - curl https://rohrli.gamlor.info/vykkev-xizras-puzwyq-vauveq | gzip -d > my-file.txt
>
>    This pipe is streaming. No data is permanently stored on the server.
>    Do not quit (Ctrl-C/D) before the transfer is done.
>
>    Verify that the download is complete and correct with: sha1sum my-file.txt
>    
>    Start your download now. We're waiting for the download to start for 10 minutes
>    ......

As you see, you get a unique link. You now can download it =). You might want to double check the download with something like sha1sum.

curl https://rohrli.gamlor.info/vykkev-xizras-puzwyq-vauveq > my-file.txt

>    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                    Dload  Upload   Total   Spent    Left  Speed
>    100     5  100     5    0     0      2      0  0:00:02  0:00:02 --:--:--     2

And you’re done!
Up and downloading of course also work with a browser =).

That’s about it. If you find a bug or have wishes, let me know =)

The post Transfer Files via Web: curl https://rohrli.gamlor.info first appeared on Gamlor.

Code Reloading in Clojure

gamlerhart — Sun, 07 Aug 2016 13:19:16 +0000

I like quick feed back while I’m coding. Instead of ‘playing’ computer in my head, let the code code run and see if it does what I expect it to do. It’s one of the reasons enjoy working in a test first workflow and having a test friendly code base.

Just changing curtains color

Anyway, some things are just easier to ‘tweak’ visually and experiment. For example when working on a GUI code or some REST output. So, I want that changes are immediately available. In Clojure there are different, ready to use solutions to do code reloading. Like Figweel for Clojurescript and web front-ends.

Here I am not going into Figweel and many other solutions. Instead I’m using Clojure facilities, without any special tools. I also take a look at on code reloader: test-referesh

Basics: Reload with use

Assuming we have some piece Clojure code. Like this:

Now, we want to ‘tune’ the email and make it actually a friendly email. So you include it into your REPL and evaluate the result:

Now you did some changes. But when you use the `create-emails` function again, it still prints out the old result. You don’t want to restart, but just reload it and keep going. Well, the ‘use’ macro already has reloading built in. Load the namespace of the file with the :reload flag:

That’s it. Clojure will reload the code in the namespace for you. Nothing else to do.

Reload more, :reload-all

Maybe the change you made is actually in some dependency. Assume we have this function in another namespace, which refers the the friendly email code:

When you use the ‘preview-email-examples’ function, you might notice that reloading with `(use ‘uses-emails :reload)`, it will not include changes made to the my-friendly-emailer.clj file. Use the :reload-all flag. It also reloads any dependencies:

Yes, please just change the curtains.

Automate Reloading, Example REST Server

You can use the ‘use’ primitive easily to create your own reload mechanism. This is useful, when you use Clojure in some embedded scenario or existing of the shelf reloaders do not work.. Let’s extend the given email example to a web service:

This web server is pretty annoying. If we change anything in the email template code, we have to restart it. Wouldn’t it be nice if changes are visible on the next browser refresh. Well, we can hack that our self. We add a developer flag. When that is set, we just go and reload our app’s main namespace.

Then we enable it. In this example just in via the REPL. In a real app the toggle mechanism can be more advanced

Of course, for Ring there is already a reload library (probably even more than one). For example the reload wrapper from ring-devel library.

Using test-refresh

So far we used core Clojure functionality. Based on that you certainly can go deeper and create more elaborate reloading schemes. One of such is test-refresh, a Leiningen plugin. I just add it to my ~/.lein/profiles.clj, so that it is available in any project I touch:

This gives the `lein test-refresh` command. That command will run the test suite. The cool thing now is, that it will rerun tests which are affected by a change. So you get immediate feed back if your edit just broke something. Even better, you just can experiment around, while test-refresh reloads the code, giving you always feed back.

Immediate feedback.

Here a screen series to illustrate that. Or if you have time, checkout this video: Don`t settle for a REPL.

When started, the tests are executed first:

When editing, as soon as you save a file, all tests which depend on that change are executed again

Now, you also just can try code out right there. The results are shown immediatly. Like a free edit REPL:

That also allows you to monitor some high level end result, while changing the lower level functions:

Use test-refresh or your Test Server

Well, you can also use test refresh to reload your server when files change. What I usually do put the server in a separate namespace, with no dependency to my core app. When the file and its dependencies does not change, test-refresh won`t reload it. This means that test refresh does not reload the actual server. Note that the test server exposes an atom for the actual request handler:

Then we add a ‘test setup’ file. That one depends on our application. Since our application changes, and this file depends on it, it will be reloaded as soon as we change anything in our applicaiton. When reloaded, it will swap out the handler of the test server. So, if the app changes, test-reload will do it`s reload logic and we also will have the latest code running in the web server:

This can be done for other server types and setups. Is neat for places where there no pre made plugin or you need a very specific setup.

Keep on reloading

That`s it for today =). Happy reloading. ^.^

The post Code Reloading in Clojure first appeared on Gamlor.

Joyent’s Triton : /native, our friend!

gamlerhart — Fri, 29 Jul 2016 12:50:01 +0000

We deployed our Docker container on Triton and everything works well. Suddenly, today there a issue. Our application is slow or stopped working. And in our container there is not debugging tool. So, what should we do?

Rebooting is no solution

Explore Triton’s Container a bit

I started again a nginx container on Triton:

When the container has started, start a shell inside the container with ‘docker exec’. Then explore the file system a bit.

/native?

What the hell is /native? Look a around a bit more. Maybe try to find popular unix tools in there? Aha! From `awk` to `zcat`, everything is there. So /native is another Unix? Right! /native is a window into Triton’s native operating system. And there are popular unix tools there.

Explore!

At /native/ old unix tools live.

Let’s use /native

Now we know that /native exists so let’s use it. First let’s deploy a bad container. I’ve prepared a ‘bad container’ example. The code is here. Let’s start that ‘gamlerhart/waste-io’ container:

This container has a issue. Let’s start finding out what the problem is. Again, we `docker exec` into the container. If we try to use `iostat`, nothing is there. However, we can use /native. Add /native to the PATH and we can use `iostat`.
.
Aha! Our container burns quite a bit of IO. However, we want to know more. Maybe some kind of tracing. Let’s dtrace. Yep, there is dtrace in /native. I’m not a dtrace pro, there is tons of materials online. (dtrace.org, guide, examples). We’ll use the ‘lx-syscall’ probe (LinuX-syscall).`-ln lx-syscall:::` lists lx-syscall probes. -n `syscall::: { @num[execname,pid,probefunc] = count() }` starts the probe, and groups by program, process-id and probe name. Stop the probe with `Ctrl+C`:

We see our node process is doing tons of write, read, open unlink and futex system calls, so our node process is the issue. But what is this program doing? What files is it opening? Let’s do more dtrace probes. `x-syscall::open:entry` only shows the open syscall, `execname==”node”` filters out node only. And we group by file with “@num[copyinstr(arg0)] = count()`.

Ah, tons of temporary files. Hmm…well where in the node program do we create these temp files? With Dtrace’s ustack/jstack we can the the programs stack trace:

Ah…no luck. The open syscall is in the event loop (Looking into node via dtrace is another topic). However, we know know that the node program creates tons of tmp/io-file files and with this information we hopefully can fix that bug.

Sherlock ‘dtrace’ Holmes

You know DTrace well?

You know dtrace well and cannot find many probes? Because you can only dtrace within the Triton container you have less probes available. And we cannot go into the global zone of Joyent’s public cloud service. On a private Triton deployment we could. That’s a topic for another time and another blog post.

The post Joyent’s Triton : /native, our friend! first appeared on Gamlor.

Hey Mr Postman, be on Every Windows Machine.

gamlerhart — Sat, 09 Jul 2016 14:04:03 +0000

Postman is probably my favorite adhoc HTTP tool for exploring API’s, testing, debugging etc etc. Often I go to some other machine, and want to test something on there. Oups: Chrome nor Postman is installed =(.
Like SSMS, I want Postman at my finger tips, on many machines. Why can’t I just type turbo run postman? That would be lovely, no?!

With V8 and Chrome to Valhalla!

Create a Postman container

Well, at the moment there is no packaged Postman on Turbo.net’s hub. But don’t worry, you can create your own image. Let’s start. First, we could start up Chrome, navigate to the store there etc: Oh wait, we can do better. On the command line, give it the URL to open up. That URL can be a Chrome store page. Just Google for “Postman Chrome Store”, copy and paste the site’s URL and there we go:

This pops open the extension add dialog. Confirm the installation.

After that, we have Chrome with Postman installed in this container. When we start this container again, Postman is installed there.

Build a Postman Image

Now, we actually want a Postman image, not Chrome with Postman installed. We can commit the container to a image and change its launch parameters to start the Postman.

Let’s do it via a Turbo script. With Turbo scripts you can script building new images. Most images on the turbo hub are actually created this way. You can look at the source for most of the images in Github https://github.com/turboapps/turbome.

Our Postman creating script will be way simpler. It can be adopted for other Chrome apps without any problems. Here it is:

Now, we can run this script to create our Postman image:

Unfortunately, I did not find out how to automate Chrome to add an extension automatically. So, you have to click ‘Add to Chrome’ and wait until it is added. Then close Chrome to continue the script.

Anyway, now you have a Postman image. Push it to the hub, so you can use it in any computer. Now you can go to any box, type in turbo run [your-account/postman into the command line, and bang, Postman will fire up.

One Postman? More Postmen!

Make It Fancy

If the image is for personal use, then you are probably already happy with the result. If you want to share this image across an organization, you might want to add some meta data. You can go to the Turbo.net hub, go to the repository of the image you just pushed. There you can add a nice icon, some explanation etc.

Adding meta data

You then can use the turbo installi/subscribe commands to add the Postman image to the start menu etc. That exercise is left for you, the reader ;). Like in my example, turbo installi roman.stoffel@gamlor.info/postman

The post Hey Mr Postman, be on Every Windows Machine. first appeared on Gamlor.