My Experience

Retry Strategy - Transient Fault Handling

2022-03-26T03:34:00.001-07:00

Systems can be susceptible to transient failures when communicating with downstream dependencies, impacting service availability, especially on the cloud, where the communication between services goes over unreliable channels.

⁣

Using a proper retry strategy is crucial to recover quickly from minor glitches, but it has to be done carefully and follows a set of best practices.

⁣

I firmly believe that no or conservative retry configuration is as bad as aggressive retries; they impact the system's availability differently.

⁣

𝗕𝗲𝘀𝘁 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗲𝘀

⁣

Every component in the call chain must have a time-to-live (Timeout) and retry configuration that works in harmony. Async workflows and batch processes should tolerate higher latency, and here a retry strategy is crucial, but it has to be limited (e.g., 3 retries).

⁣

Systems should have a mechanism to throttle bursts in traffic to avoid retry storms, which may impact the system's recovery during major outages.

⁣

Using exponential back-off with jitter strategies when configuring retries, the waiting time should exponentially increase (e.g., First retry 100ms, second after 200ms, and third after 500ms).

⁣

Back-off only isn't very helpful as retries of multiple failed requests simultaneously cause contention and overload again, so adding randomness (Jitter) to the waiting time (back-off) spreads the retries of failures around in time.

⁣

Retries may impact latency, and you have to measure its impact on the end-to-end flow; control-plane vs. data-plane react differently to such effect; hence it's up to you to decide the best retry strategy for your usecase.

⁣

Overly aggressive retry strategies (Too short intervals or too frequent retries) can adversely affect the downstream service recovery during major outages.

How do I move to the next level?

2022-02-15T05:00:00.002-08:00

If you are wondering how you could move to the next level, Junior to Senior level, or Senior to a Tech Lead level. Here is an idea!

Take some of the ideas I post here from time to time. I will try to post something every week, compile a small document that briefly describes the initiative and its benefits, and present it to your team.

You don't have to implement it entirely, but you could become the go-to person for this topic in your team from now on ;)

I'd also go above and beyond and create a backlog story with refined sub-tasks that are ready for pick-up in future sprint plannings!

If you are a Senior-level, I think it's more convenient to use such ideas as onboarding projects for new hires; it will be a great onboarding experience ;)

Standard Operating Procedure (SOP)

2022-02-13T08:32:00.002-08:00

𝐒𝐭𝐚𝐧𝐝𝐚𝐫𝐝 𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐧𝐠 𝐏𝐫𝐨𝐜𝐞𝐝𝐮𝐫𝐞 (𝐒𝐎𝐏)

In some companies, every team is solely responsible for the design and development of their systems and what comes after that, such as deployments, monitoring, and troubleshooting of errors.

An SOP document is crucial to provide detailed step-by-step instructions to complete specific processes.

For example, when a failure occurs (Alarm A is triggered), a team member should find the steps to troubleshoot and mitigate this error in the SOP doc!

𝐆𝐨𝐨𝐝 𝐒𝐎𝐏

Go to logs for stage X (Link)
Check the dashboard for stage X (Link)
Update the ticket with what you found in the logs, and a link to the graphs
Ensure the Dead Letter Queue (DLQ) is empty; if not, apply the SOP section for retrying DLQ messages (Link)
Check if there is a customer impact (usually customer X, Y, and Z are impacted by such error) - (Link to clients dashboard)
Check if the error is because of a downstream dependency (Link to dependency H and R dashboards)
Update the ticket with your findings above, and engage on-calls for client/dependency teams if necessary.
....

𝐖𝐡𝐞𝐧 𝐝𝐨 𝐭𝐞𝐚𝐦𝐬 𝐝𝐞𝐟𝐢𝐧𝐞 𝐒𝐎𝐏𝐬?

Unfortunately, most teams write the SOP sections for specific errors only after they occur, but in our team, we started to change our strategy and always define all possible points of failures during design and development, and add tasks to update the SOP doc with a section for each failure type!

For existing systems, I highly recommend reading about Pre-Mortem process.

Technology Evolves, Facts can change too?

2022-02-13T08:29:00.000-08:00

In the past few weeks, I was involved in several discussions about topics that I had always thought they are over-studied, and pretty much everyone is aligned around them!

I had the impression, why are we discussing this now? - there are tons of articles and books citing case studies around them, and they all reached a similar conclusion

Then, I decided to take a step back and listen carefully to the other person’s insights and reasoning that made him reach such conclusion and thought!

The thoughts and arguments were good, they weren’t strong enough to fully convince me, but if you think about it from a different perspective, you may find it crucial to continue challenging those thoughts, opinions, and facts from time to time.

Technology has advanced so much, and businesses have crazily evolved recently. So maybe those facts that made sense ten years ago aren’t relevant anymore.

End of text

Modulo Operation Tricks

2020-06-21T11:34:00.001-07:00

When you solve a problem that might result in values beyond integer limits (e.g., Wouldn't fit into a 64bit integer), you're often asked to print the value modulo a significant prime value (e.g. 10^9 + 1).

To be able to use the modulo operation correctly, you need to understand its properties.

Trick #1 Multiplication/Summation of values

For example, if you are multiplying/adding a set of numbers (a, b, c, d) then its useful to know that:

(a*b*c*d)%N = (a%N)*(b%N)(c%N)*(d%N) = ((a*b*c)%N)*d%N

The above is quite useful when you want to always keep calculations to fit into the given limits.

Trick #2 Multiplicative Inverse

For division, it's a little bit tricky though, you can't merely assume that:

(a/b)%N = (a%N/b%N)

Although that would be awesome, it's unfortunately not correct - So to solve this, you have to understand the modular multiplicative inverse of a number A.

So to calculate (a/b)%N, you can convert it to a%N*power(b, N-2) - I will not explain how or why, please refer to the wiki page above for more details.

Trick #3 Negative Numbers

Another problem one would face during programming contests is when you calculate modulo for a negative number, which doesn't really work as intended.

So -1%10 would result in -1 rather than 9 - This really depends on the programming language you use but this trick is useful for Java and C++ but not for Python which produces the right value 😊.

To overcome this, you can calculate it by adding N to the number so, -1%10 = 9%10 = 9 👍.

That's it! I will keep this post up to date with any other tricks I find in the future.

Speak so that others want to listen

2020-05-24T09:27:00.001-07:00

How many times were you in a meeting or discussing with a group of people, and was your turn to say something, and someone either interrupted you or waited until you finished and continued talking or just switched the topic?

I'm pretty sure this has happened many times throughout your career and will continue to happen even after you read this article and practice its techniques, but I can promise it will be less often than before.

I've been observing others and myself in meetings and always found out that most of the discussions followed the same scenario I've described above, so I started to look for ways to improve the way I speak and spotting the mistakes I and others often do.

I've found many techniques and suggestions that are helpful for someone to speak effectively and make people more engaged to what he/she says, I've chosen only 3 of them to talk about in this article.

Note: The following techniques are not only useful when speaking, but for example, rule #3 is a handy technique to write persuasive and easy to follow/read content (e.g., Email).

1. The power of the pause

This technique is one of the essential methods one could learn and practice to improve the effectiveness of his contributions in discussions and meetings.

Adding a pause before you talk allowing the other person to finish, rather than jumping with the first thing comes to your mind to avoid the risk of interrupting the other person if he wants to continue to add something.

Adding pauses when you jump from one point to the next, grabs your audience full attention to what you are saying.

This technique will make people feel more comfortable talking to you and, of course, actively listening to you and what you say. It will also make you and the others more engaged in the discussion.

2. Speak with clarity and confidence

How many times have you been asked (or asked someone) to repeat what you/he/she's just said (multiple times)?

One mistake we all do (both native and non-native speakers) is speaking too fast and tend to swallow the endings of some words, which might change the meaning or cause confusion. If the audience lost focus for few seconds to think about what you said or feels difficulty tracking it, then he will either ask you to repeat what you just said or stop focusing because he is still thinking about some words that you mispronounced.

So next time, try to speak at a slower pace and make yourself clear when speaking. You will see the difference in how confident you become and how engaged the audience is - practicing this technique will improve your cadence, and later it will naturally happen when you speak.

One tip I liked is to speak slowly as if you are giving your phone number to someone and want to make sure he gets it right and able to write it down.

3. The rule of three

Structuring your content, for example, when writing an Email or in a presentation in three parts, makes it very easy for your audience to follow and remember and makes you (the author) as a speaker or writer appear more knowledgeable, credible, and convincing.

As you can see, I followed this approach in this article, and after reading through it, I genuinely believe that you will be able to memorize all three techniques quickly next time you speak or write about something.

Do you know more techniques? Please comment on the article and share them with other readers and me. And as a little practice, write your review using the above methods about the topic and share it with friends.

The Beatles Problem A - Codeforces #549 DIV1

2019-04-06T17:39:00.001-07:00

Few days ago I was trying to solve the beatles problem which was a little bit tricky and I had to analyze it to find enough observations in order to come up with a good solution .. In this article I will go through my observations step by step and hope its easy for you to digest.

The editorial of the problem is very concise and not easy for everyone to understand so I will be as detailed as possible in this article to make sure you get the idea by going through my observations one by one and of course combine them to code a solution.

Observation #1:

Lets assume the jump we take each time is L, how many jumps do we need to get to S again (S is the start position)? Lets find out by example:

Note: For this observation we can completely ignore a and b

Assume we have n = 2, k = 3 and we start at S = 2:

If L = 1 then the path will be {2, 3, 4, 5, 6, 1, 2} => 6 jumps!

If L = 2 then the path will be {2, 4, 6, 2} => 3 jumps!

If L = 3 then the path will be {2, 5, 2} => 2 jumps!

From the above example we figured out that #jumps = #cities/gcd(#cities, L)

Where gcd is Greatest Common Divisor.

Observation #2:

We need to figure out L as its much easier to use to brute force a solution (I will show you why in the next observation).

To be honest I didn't get to that observation until I took a small hint from a solution which I couldn't understand (The solution was checking 4 possible values to find L) :O but after some manual work I figured it out.

The 2 X markers on the line are the first and second cities .. from the problem statement we know that city1 has the nearest restaurant at distance A and city2 has nearest restaurant at distance B -> Nice!

We have 2 possibilities for nearest restaurant of the first city .. either its before or after (marked as A1, A2) .. Same for nearest restaurant of the second city (marked as B1, B2).

We know also from the problem statement that the distance between any two neighboring restaurants = K, That means the distance between A* and B* is i*k, where i is any value between 1 ... n (We will know why its < n in the next observation).

- Remember we need to figure out the distance between the 2 Xs which is ???

- Yes, correct - its L

That leaves us with the following 4 possibilities:

Possibility #1: Restaurants are located at (A1, B1):

-> L = i*k + (B1 - A1)

Possibility #2: Restaurants are located at (A1, B2):

-> L = i*k + (-B2 - A1)

Possibility #3: Restaurants are located at (A2, B1):

-> L = i*k + (A2 + B1)

Possibility #4: Restaurants are located at (A2, B2):

-> L = i*k + (A2 - B2)

Observation #3:

Now we need to iterate to find the value of L, but what are the boundaries to find L?

We know that L = i*k + c, where c is one of the possible values {(A+B), (A-B), (-A-B), (B-A)}.

We also know we have only N*K cities .. so i could be any value between (1 ... N), right?

Now with the above 3 observations we can easily code a solution .. here is my C++ solution for this problem.

#include<iostream>
#include<map>
#include<set>
#include<algorithm>
#include<cstring>
using namespace std;

main() {
        long long n, k, a, b;
        cin >> n >> k >> a >> b;

        long long possible[] = {a+b, a-b, -a-b, b-a};

        long long mn = n*k, mx = 1;

        for (int i = 1; i <= n; i++) {
                for (int j = 0; j < 4; j++) {
                        long long x = possible[j]%k;
                        if (x < 0) x += k;

                        mn = min(mn, n*k/__gcd(n*k, i*k+x));
                        mx = max(mx, n*k/__gcd(n*k, i*k+x));
                }
        }

        cout << mn << " " << mx << endl;
}

If you have questions/feedback or spot a mistake please let me know in the comments.

Java ForkJoinPool with an exmaple

2019-01-13T16:12:00.000-08:00

ForkJoinPool was introduced with the release of Java 7 to solve a very particular set of problems that tend to be hard to solve with any other thread pool implementation.

The class is designed to work with divide and conquer algorithm: those where a task can recursively broken into broken sub-tasks.

It looks just like any other thread pool e.e.,ThreadPoolExecutor class, it implements the Executor and ExecutorService interfaces. It uses an unbounded list of tasks that will be run by the number of worker threads configured or by default the number of CPUs exist.

Example: Parallel Merge Sort

Sorting an array of 1 million elements. We have 3 main sub-tasks to sort the array:

Sort the first half of the array
Sort the second half of the array
Merge the two sorted sub-arrays

The base case is when its faster to use insertion sort to sort the sub-array (lets assume when the array has 10 elements) of course makes more sense than using parallel merge sort here. In the end there will be 1 million tasks to sort the leaf arrays, more than 500,000 tasks are needed to merge those sorted sub-arrays, and more than 250,000 tasks to sort the next merged sub-arrays .... and so on.

The most important point to notice is that none of the tasks can complete until the tasks that they have spawned have also completed. Here is when the ForkJoinPool comes very handy. Of course its doable through a ThreadPoolExecutor but can't be done as efficient as ForkJoinPool and the implementation is much more complex.

In ThreadPoolExecutor a parent task must wait for its child tasks to complete, A thread cannot add another task to the queue and then wait for it to complete as once a thread is waiting it can't be used to run one of the sub-tasks.

bla bla bla ... show me the code.

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;

import java.util.Arrays;
import java.util.concurrent.ForkJoinPool;
import java.util.concurrent.RecursiveTask;
import java.util.concurrent.ThreadLocalRandom;
import java.util.stream.IntStream;
import lombok.AllArgsConstructor;

@AllArgsConstructor
public class ForkJoinPoolSample {

  private static int[] elements;

  @AllArgsConstructor
  private static class MergeSortTask extends RecursiveTask<Integer> {
    private int first;
    private int last;

    @Override
    protected Integer compute() {
      int len = 0;

      if (last-first <= 10) {
        Arrays.sort(elements, first, last+1); 
        len = (last-first+1);
      } else {
        /* Sort two sub-arrays */
        int mid = (first+last) >>> 1;
        MergeSortTask leftSubtask = new MergeSortTask(first, mid);
        leftSubtask.fork();
        MergeSortTask rightSubtask = new MergeSortTask(mid+1, last);
        rightSubtask.fork();

        len += leftSubtask.join();
        len += rightSubtask.join();

        /* Merge two sorted sub-arrays */
        MergeTask mergeTask = new MergeTask(first, mid, last);
        mergeTask.fork();
        mergeTask.join();
      }

      return len;
    }
  }

  @AllArgsConstructor
  private static class MergeTask extends RecursiveTask<Integer> {
    private int first;
    private int mid;
    private int last;

    @Override
    protected Integer compute() {
      int[] tmp = new int[last - first + 1];
      int left = first, right = mid + 1, indx = 0;

      while (left <= mid && right <= last) {
        if (elements[left] <= elements[right]) {
          tmp[indx++] = elements[left++];
        } else {
          tmp[indx++] = elements[right++];
        }
      }

      while (left <= mid) {
        tmp[indx++] = elements[left++];
      }

      while (right <= last) {
        tmp[indx++] = elements[right++];
      }

      for (indx = 0; indx < tmp.length; indx++) {
        elements[first + indx] = tmp[indx];
      }

      return tmp.length;
    }
  }

  private static void createRandomInts() {
    elements = new int[100000];
    final ThreadLocalRandom random = ThreadLocalRandom.current();

    IntStream.range(0, 100000)
        .forEach(i -> elements[i] = random.nextInt());
  }

  public static void main(String[] args) {
    createRandomInts();

    long before = System.currentTimeMillis();
    int n = new ForkJoinPool().invoke(new MergeSortTask(0, elements.length-1));
    long after = System.currentTimeMillis();

    System.out.println("Sorted " + n + " Elements in " + (after-before) + " ms.");

    boolean sorted = IntStream.range(0, elements.length-1)
        .allMatch(i -> elements[i] <= elements[i+1]);

    assertThat(sorted, is(true));
  }
}

From the doc:

fork(): Arranges to asynchronously execute this task in the pool the current task is running in

join(): Returns the result of the computation when it is done

Those methods use a series of internal, per-thread queues to manipulate the tasks and switch threads from executing one task to executing another. Of course all of that is transparent to the developer.

References

ForkJoinPool documentation

RecursiveTask documentation

ForkJoinTasks documentation

Java Performance The definitive Guide

Java Weak References

2019-01-05T12:54:00.000-08:00

Reusing objects is important in Java but also it can cause memory and performance issues if the objects to be reused can't be freed out by GC. Sometimes we need to reuse objects as soon as they are still being referenced or as soon as they may have good chance to be used in the future.

Weak references give us the opportunity to achieve this by not largely affecting memory and performance of our application (In a more GC-friendly). In today's post I will go through the differences between strong and weak references in Java and the effect of them on garbage collector.

Strong References

Strong references are the default type/class references in Java. Objects that are strongly references aren't eligible for GC neither minor nor full GC until they are not strongly referenced anymore.

All object references in Java are strongly referenced unless explicitly specified, see below sections.

StringBuilder builder = new StringBuilder();

Weak References

Weak reference objects are not default and must be explicitly specified. This type of references is to maintain references to objects that are `WEAK` meaning that if the object is eligible for GC `not strongly referenced anymore` but still `weakly referenced` the object can be collected.

StringBuilder builder = new StringBuilder();
WeakReference<StringBuilder> weakBuilder = new WeakReference<>(builder);

Soft References

This type of references is also a weak reference that remains in memory for longer - It resists minor GC until memory is really needed to be reclaimed (Application is reaching OOM then it will clear all soft references). Soft references are essentially one large, least recently used (LRU) object pool (cache).

The JVM keeps track of the last access to each reference and calculates if the soft reference is eligible for GC. This is controlled by the JVM `-XX:SoftRefLRUPolicyMSPerMB` flag which is by default = 1000 = One second of lifetime per free megabyte in the heap

StringBuilder builder = new StringBuilder();
SoftReference<StringBuilder> weakBuilder = new SoftReference<>(builder);

Phanotm References

Phantom references help us doing finalization to avoid implementing the `Object.finalize` method which could have negative implications to the application as its not deterministic and can make the object reachable again or negatively impact application performance.

Phantom references are different from Weak/Soft references in that GC will not collect a phantom reachable object until its cleared out all acquired resources.

`PhanomReference` class accepts two parameters the referent object and a `ReferenceQueue` which is then used to enqueue objects eligible for clearance.

ReferenceQueue q = new ReferenceQueue();
PhantomReference pr = new PhantomReference(object, referenceQueue);
// Later on another point
Reference r = q.remove();
// Now, clear up any thing you want

Quick Summary (From: Java Performance - The Definitive Guide)

Indefinite (soft, weak, phantom) references alter ordinary lifecycle of java objects, allowing them to be reused in ways that may be more GC-friendly.
Weak references should be used when an application is interested in an object only if that object is strongly referenced else where in the application.
Soft references hold onto objects for (possibly) long periods of time, providing a simple GC-friendly LRU cache.
Indefinite reference consume their own memory and hold onto memory of other objects for long periods of time; they should be used sparingly.

Command line tools every (Java) developer should know

2018-12-24T13:35:00.000-08:00

In this post I will go through some useful command line tools to debug and tune your service. I will show a sample output and description of the most important parts of the output, feel free to read the manual pages for more details about the tools and supported flags to tune their output.

vmstat

This system tool is very useful to view and monitor statistics of your system processes, memory, paging, block IO, disks, and CPU. Its useful to check IO/CPU utilization and spot any bottlenecks.

Example:

$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 12583088 119480 1590740    0    0   406    72  470 1035 10  3 87  0  0
 0  0      0 12582832 119480 1590740    0    0     0     0  451 1562  1  0 98  0  0
 0  0      0 12579484 119480 1590732    0    0     0     0  408 1345  1  0 98  0  0

The above command is to show stats with 1 second delay.

Processes: Amount of runnable and un-interruptible processes

Memory: Free/cache memory and memory used as buffers and the amount of virtual memory used (swpd)

Swap: Swap in and out (from disk)

IO: Blocks in (READ) and out (WRITE) from disk

System: Number of interrupts per second and context switches

CPU: Time running user code vs system (kernel) code, idle time, and waiting for IO

iostat

This system tool is very useful to view and monitor IO statistics of your system.

Example:

$ iostat -xm 5
Linux 4.15.0-42-generic

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.27    0.02    1.59    0.15    0.00   92.97

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     5.29   20.82    6.42     0.43     0.11    40.69     0.02    1.18    0.25    4.20   0.14   0.39

The -x flag is to show extended statistics and -m is to view stats in megabytes per second. Of course some of the information above can also be viewed by the previous tool `vmstat`.

%user, %system, %iowait, %idle: The percentage of CPU spent running user code, system code, waiting for IO, and idle

rrqm/s, wrqm/s: The number of (read/write) requests merged per second that were queued to the device

r/s, w/s, rMB/s, wMB/s: Reads/Writes (after merge) per seconds, MBs per seconds (read/write)

await, r_await, w_await: Average time in ms for IO requests to the device to be served, average time in ms for read, average time in ms for write

nicstat

This tool isn't installed by default in linux systems but can be installed using package managers or downloaded from sourceforge. It basically does the same as previous tools but for network monitoring.

Example:

$ nicstat 5
    Time      Int   rKB/s   wKB/s   rPk/s   wPk/s    rAvs    wAvs %Util    Sat
20:10:52       lo    0.33    0.33    3.98    3.98   85.25   85.25  0.00   0.00
20:10:52   wlp2s0    6.18    1.85   12.23    9.03   517.6   210.0  0.00   0.00

Prints network statistics ever 5 seconds.

Int: Interface name

rKB/s, wKB/s: KBs Read(Received) Write(Sent)

rPk/s, wPk/s: Packets received/sent

rAvs, wAvs: Average packet size received/sent

%Util, Sat: Percentage utilization of the interface, Saturation (errors/second)

jcmd

This tool and the coming few ones come along with the JDK and can be used to send diagnostic commands to a running JVM.

Example:

$ jcmd 8880 VM.uptime # 8880 is the process ID, VM.uptime is to check uptime of the running java process

8880:
163.883 s

$ jcmd 8880 VM.system_properties # View system properties of a single java process

8880:
#Mon Dec 24 20:32:01 CET 2018
java.runtime.name=OpenJDK Runtime Environment
sun.boot.library.path=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64
java.vm.version=25.191-b12
java.vm.vendor=Oracle Corporation
java.vendor.url=http\://java.oracle.com/
path.separator=\:
java.vm.name=OpenJDK 64-Bit Server VM
file.encoding.pkg=sun.io
user.country=US
..... # and more .....

$ jcmd 8880 VM.version

8880:
OpenJDK 64-Bit Server VM version 25.191-b12
JDK 8.0_191

$ jcmd 8880 VM.command_line # The command line of the process was started

8880:
VM Arguments:
java_command: HelloWorld -Dse7so=29 -client
java_class_path (initial): .
Launcher Type: SUN_STANDARD

$ jcmd 8880 VM.flags # Enable flags - try with [-all]

8880:
-XX:CICompilerCount=3 -XX:InitialHeapSize=262144000 -XX:MaxHeapSize=4175429632 -XX:MaxNewSize=1391460352 -XX:MinHeapDeltaBytes=524288 -XX:NewSize=87031808 -XX:OldSize=175112192 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseParallelGC

$ jcmd 8880 Thread.print # Thread dump - see also jstack tool below
# Very long output here that can be saved to a file for later analysis

$ jcmd 8880 GC.run #
8880:
Command executed successfully

$ jcmd 8880 GC.class_histogram # Show heap histogram ([C is character array, [B is byte array, [I is integer array)
8880:

 num     #instances         #bytes  class name
----------------------------------------------
   1:          1157        1614520  [C
   2:           131         216072  [I
   3:           474          53992  java.lang.Class
   4:          1145          27480  java.lang.String
   5:           531          26632  [Ljava.lang.Object;
   6:            20          25264  [B
   7:           378           9072  java.util.LinkedList$Node
   8:           211           6752  java.util.HashMap$Node
   9:           190           6080  java.util.LinkedList
  10:            75           5400  java.lang.reflect.Field
  11:           256           4096  java.lang.Integer
  12:            93           3720  java.lang.ref.SoftReference
  13:           111           3552  java.util.Hashtable$Entry
  14:             7           2632  java.lang.Thread
  15:            12           1984  [Ljava.util.HashMap$Node;
  16:            51           1896  [Ljava.lang.String;
  17:            38           1824  sun.util.locale.LocaleObjectCache$CacheEntry
  18:            47           1504  java.util.concurrent.ConcurrentHashMap$Node
  19:            23           1472  java.net.URL
  20:            14           1120  [Ljava.util.WeakHashMap$Entry;
  21:            14           1120  java.lang.reflect.Constructor
  22:             2           1064  [Ljava.lang.invoke.MethodHandle;
  23:             1           1040  [Ljava.lang.Integer;
  24:            26           1040  java.io.ObjectStreamField
  25:             6            992  [Ljava.util.Hashtable$Entry;
  26:            15            840  java.lang.Class$ReflectionData
  27:            21            840  java.lang.ref.Finalizer

jconsole

In fact this tool is a GUI tool but it worth mentioning as it provides very useful information to monitor and control java applications.

It views information about Memory, CPU, Threads, and Classes loaded.

jmap

Provides heap dumps and other information about JVM memory usage.

Example:

$ jmap -dump:live,format=b,file=/tmp/dump.hprof 9293
Dumping heap to /tmp/dump.hprof ...
Heap dump file created

Generates a dump from a live process, in binary format and saves it to /tmp/dump.hprof - you can then analyze it with any heap dump analyzer.

jstack

Useful to generate thread dumps with full stack traces.

Example:

$ stack -l > /tmp/myapp.tdump

You can then analyze the thread dump using any thread dump analyzer.

visualvm

This is also a GUI tool but very useful to monitor your java application. Its man page describes it as all-in-one java troubleshooting tool and in fact it is very nice and can be very useful along with the above commands. you can also use to analyze heap and thread dumps generated earlier using jmap and jstack.

jstat

Is one of the best tools to view very useful stats about GC, JIT compiler, and so many other things to monitor.

Example:

$ jstat -gcutil -t 9293 100ms
Timestamp         S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT   
         5452.8   0.00   0.00  77.88   0.86  96.16  84.57     10    0.056    10    0.295    0.351
         5453.0   0.00   0.00  77.88   0.86  96.16  84.57     10    0.056    10    0.295    0.351

The above shows GC information (the -t to show timestamps) every 100ms.

S0, S1: Survivor spaces 0 and 1 utilization as a percentage of the space's current capacity

E, O, M: Eden space, Old space, Metaspace utilization as percentage of the space's current capacity

CCS: Compressed class space utilization as a percentage

YGC, YGCT: Number of young generation GC events and young generation gc time

FGC, FGCT: Number of full GC events, and full GC time

GCT: Total garbage collection time

When the Eden space percentage goes up this triggers a young generation minor GC event which then will reduce its size and the S0, S1 will be swapped and of course Old generation increases a bit (some objects move from Y to O) - Full GC events are triggered when the Old generation percentage goes up and as a result its size is reduced and fragmented.

$ jstat -class -t 9293 1s
Timestamp       Loaded  Bytes  Unloaded  Bytes     Time   
         5845.2   3602  7665.3       13    15.8       1.33
         5846.2   3602  7665.3       13    15.8       1.33
         5847.2   3602  7665.3       13    15.8       1.33
         5848.2   3602  7665.3       13    15.8       1.33

The same command but to view class loader statistics every 1 second.

Loaded: Number of classes loaded

Bytes: Number of bytes loaded

Unloaded: Number of classes unloaded

Bytes: Number of bytes unloaded

Time: Time spent performing class loading and unloading

$ jstat -printcompilation -t 9293 1s
Timestamp       Compiled  Size  Type Method
         6067.7     3744    684    1 java/util/ComparableTimSort sort
         6068.7     3744    684    1 java/util/ComparableTimSort sort
         6069.7     3744    684    1 java/util/ComparableTimSort sort

The same command but to view Java HotSpot VM compiler method statistics.

Compiled: Number of compilation tasks performed on the method

Size: Size of code byte code (not binary code)

Type: Compilation type of the most recently compiled method

Method: Class name and method name identifying the most recently compiled method

$ jstat -compiler -t 9293 1s
Timestamp       Compiled Failed Invalid   Time   FailedType FailedMethod
         6378.9     3753      3       0    12.54          1 org/netbeans/lib/profiler/server/ProfilerRuntimeObjLiveness traceObjAlloc
         6380.0     3753      3       0    12.54          1 org/netbeans/lib/profiler/server/ProfilerRuntimeObjLiveness traceObjAlloc
         6381.0     3753      3       0    12.54          1 org/netbeans/lib/profiler/server/ProfilerRuntimeObjLiveness traceObjAlloc
         6382.0     3753      3       0    12.54          1 org/netbeans/lib/profiler/server/ProfilerRuntimeObjLiveness traceObjAlloc

Java HotSpot VM Just-in-Time compiler statistics.

Compiled: Number of compilation tasks performed

Failed: Number of failed compilation tasks

Invalid: Number of invalidated compilation tasks

Time: Time spent on compilation

FailedMethod: The most recent failed method to compile

The last two examples are a bit hard to understand if you don't really know the basics of JIT and how it works .. you can go ahead and read what happens behind the scene to realize the importance of both examples and how you can use them to diagnose possible performance issues in your services.

Those are the most useful tools from my point of view to monitor, diagnose, troubleshoot your services .. please comment with other tools you might see useful but not stated above.

Cheers

The guide to be a better team member

2018-10-20T12:54:00.000-07:00

Its been a while since I posted something, and even longer for a non-tech one.

Today's post is a recipe for being a better team member or even better if you already are. The major thing here is to start applying all the following steps yourself before asking others to do them.

Step 1: Feedback

Everyday we give or receive feedback in different formats (e.g., Code review, Design review, or even face to face feedback). I see most of the issues that arise in every team is because of feedback - either someone gives feedback in an inappropriate way or not being open for feedback.

My manager once told me "Any feedback is a good feedback", and another manager told me once "Take the feedback and filter what really makes sense to you but always respect and assume good intentions".

The idea behind this is to always assume good intentions keeping in mind culture difference and preferences of people based on culture, background, and experience.

In case you are the one who is giving feedback, always think about it and think what is the best way to give it either in private or if OK to be in public (e.g., Review sessions). Also keep in mind culture difference and other people's preferences and make sure you don't turn people down by giving feedback inappropriately, this way you will lose them or even worse lose their trust.

Step 2: Disagree and commit

Everyday you and other team members propose solutions for current team problems either technical or non-technical. You and your colleagues have less chance to agree on all of them.

Never ever take it personal or be frustrated that there are different opinions or views "I know its hard" .. but usually if the team agrees on something and commit to it, you MUST commit to it as well. This has great potential for someone in the team to learn something new.

Step 3: Choose a mentor for yourself

Mentoring is one of the best ways for someone to learn and receive great feedback that helps improving.

I personally usually ask one or two of my colleagues to observe me and give me feedback in private if they have any, from my personal experience this helped me to improve in different areas which I had good potential for improvement "and of course still have".

Step 4: Mentor someone

This is exactly the same as step number 3 but the other way around, where you will be the mentor/observer for another team memer(s).

From my experience this helps earning your colleagues' trust and appreciation for your honest feedback both positive and non-positive.

Step 5: Learn from your mistakes

This is of course the clearest one so far. Its also a great way to measure improvement.

As an example if you receive code review feedback in an open way as potential to learn something new by constructive discussions with other team members, only this way you will learn and remember to not repeat the same mistake again and again.

Good luck

AWS kubernetes + Mongodb cluster network configuration

2018-04-01T11:49:00.000-07:00

In this tutorial I will quickly go through the steps I have done to set up two different clusters and properly configure both of them in a way that suits my need .. here are what I wanted to have in the end:

- Setting up a kubernetes cluster with one master and 2 nodes

- Setting up a mongodb cluster with one primary and 2 secondary instances

- Be able to connect from my application running on kubernetes to mongodb servers

- Mongodb server to not be publicly exposed .. so I wanted to keep it private for security reasons

Ready? Lets go ;)

1. Create kubernetes cluster using kops

Note: If your cluster is not created successfully .. may be you need to request some limits of instance types you configure for master and nodes of the cluster ;)

2. Create mongo-db cluster

I just went through this awesome guide which is using a basic AWS template to create all needed resources and configuration.

3. Connect both clusters together

Now at this step you can easily SSH to your bastian instance and connect to your mongodb service and test your credentials and so on.

Lets try to connect from our publicly available bastian server:

se7so@se7so:~/Desktop/se7so-workspace$ ssh -i "mykeypair.pem" ubuntu@ec2-bastian-****
Last login: Sun Apr  1 17:15:28 2018 from *.*.*.*
ubuntu@linux-bastian:~$ ssh -i "mykeypair.pem" ec2-user@mongodb-primary
Last login: Sun Apr  1 17:15:28 2018 from *.*.*.*
[ec2-user@mongodb-primary ~]$ mongo
MongoDB shell version v3.4.14
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.14

But you may want to be able to directly connect to your DB from your application which will later run as a container on K8s POD.

But in this template the mongodb has only a private IP and not exposed publicly through a dns or elastic IP .. it also belongs to a diff VPC than our K8s cluster ...

You will need to configure two things in order to successfully connect to your mongodb instance from your K8s cluster.

3.1 Create a connection peering between both VPCs (Make sure you pick up the VPC ID that your mongodb instances belong to, and the one that your K8s instances belong to).

3.2 Once requested .. you will need to accept the request, right click accept connection peering.

3.3 Once you accept the request .. you will need then to adjust the route tables of both VPCs by adding a new route (Destination should be the CIDR of other VPC, and target should be the ID of the connection peering request you created earlier).

At this step you configured both VPCs to be able to connect the IP ranges from the other one .. but that's not enough to connect to our mongodb cluster either through SSH or mongodb port(s).

3.4 Go to security groups of your mongodb primary instance and edit the inbound rules by allowing the security group of K8s cluster instances to connect on mongodb ports, you can also allow SSH.

Note: Outbound is accepting all traffic by default so no need to change anything.

4. Connect from K8s cluster

Now you can SSH to any of your K8s nodes or master and SSH to mongodb primary instance using its private IP.

root@k8-master:/# telnet 10.*.*.* 27017
Trying 10.*.*.*...
Connected to 10.*.*.*.
Escape character is '^]'.

5. Test connection from a running POD on K8s

Now create a kubernetes pod and SSH to it, to test your connection to mongodb instance.

root@nginx-7c87f569d-9mp8q:/# telnet 10.*.*.* 27017
Trying 10.*.*.*...

Oops it doesn't work :S

After many tries and alot of research I found some solutions to manually adjust iptables rules to route traffic to IP ranges of mongodb cluster to the host machine .. but I didn't like this solution as its not sustainable enough for me..

Now its time to use my kubernetes tricks :) .. what I came up with is the following ;)

5.1 Create a kubernetes service without a selector .. to point to the mongodb IP/PORT .. here are my configuration:

service.yaml

---
kind: Service
apiVersion: v1
metadata:
  name: my-service
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 27017

endpoints.yaml:

---
kind: Endpoints
apiVersion: v1
metadata:
  name: my-service
subsets:
  - addresses:
      - ip: 10.0.4.205
    ports:
      - port: 27017

Run:

se7so@se7so:~/Desktop/se7so-workspace/$ kubectl apply -f .
endpoints "my-service" created
service "my-service" created

SSH again to your K8s POD and run:

se7so@se7so:~/Desktop/se7so-workspace/$ kubectl exec -it nginx-7c87f569d-9mp8q bash
root@nginx-7c87f569d-9mp8q:/# telnet my-service.default.svc.cluster.local 80
Trying 100.71.103.125...
Connected to my-service.default.svc.cluster.local.
Escape character is '^]'.

Using mongo CLI:

Note: I'm redirecting 80 to 27017 ports so don't get confused .. not sure why I did it this way :P

root@nginx-7c87f569d-9mp8q:/# mongo --host my-service.default.svc.cluster.local --port 80
MongoDB shell version: 3.2.11
connecting to: my-service.default.svc.cluster.local:80/test
Welcome to the MongoDB shell.
s0:PRIMARY>

Note: you don't have to use `.default.svc.cluster.local` postfix, you can just type 'my-service'

Pingo it works and its more sustainable solution ;) .. I really hope this will help you saving some hours of research.

Kubernetes basics with demo

2017-04-28T13:17:00.001-07:00

Now we have come to a point where we can deploy and manage our deployments through Kubernetes .. if you haven't read previous two posts please read them Docker + Microservices all in one and Create/Manage docker swarm cluster.

I will start the topic by covering some of the basic concepts and keywords in the kubernetes world .. for more detailed articles please refer to the kubernetes interactive tutorial here.

Kubernetes

Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way

A Kubernetes cluster consists of two types of resources:

The Master coordinates the cluster

Nodes are the workers that run applications

Kubernetes Node

A node is a VM or a physical computer that serves as a worker machine in a Kubernetes cluster

Each node has a Kubelet, which is an agent for managing the node and communicating with the Kubernetes master

The node should also have tools for handling container operations, such as Docker or rkt

A Kubernetes cluster that handles production traffic should have a minimum of three nodes.

Kubernetes Master

Is responsible for managing the cluster.

The master coordinates all activities in your cluster, such as:

Scheduling applications

Maintaining applications' desired state

Scaling applications

Rolling out new updates

Kubernetes Deployment

The deployment is responsible for creating and updating instances of your application
Once you have created a deployment, the master schedules the application instances that the deployment creates onto individual nodes in the cluster
Once the application instances are created, a kubernetes deployment controller continuously monitors those instances
If the node hosting an instance goes down or is deleted, the deployment controller replaces it .. this provides a self healing mechanism to address machine failure and maintenance

PODS

A POD is a kubernetes abstraction that represents a group of one or more application containers (such as docker, rkt), and some shared resources for those containers such as:
- Shared storage, as volumes
- Networking as a unique cluster IP address
- Information about how to run each container, such as the container image version or specific ports to use

Summary

A Pod always runs on a Node
A Node is a worker machine in Kubernetes and may be either a virtual or a physical machine, depending on the cluster
Each Node is managed by the Master
A Node can have multiple pods, and the Kubernetes master automatically handles scheduling the pods across the Nodes in the cluster
The Master's automatic scheduling takes into account the available resources on each Node
Every Kubernetes Node runs at least:

Kubelet, a process responsible for communication between the Kubernetes Master and the Nodes; it manages the Pods and the containers running on a machine
A container runtime (like Docker, rkt) responsible for pulling the container image from a registry, unpacking the container, and running the application

Kubernetes Pods are mortal. Pods in fact have a lifecycle
When a worker node dies, the Pods running on the Node are also lost.
A ReplicationController might then dynamically drive the cluster back to desired state via creation of new Pods to keep your application running
Once you have multiple instances of an Application running, you would be able to do Rolling updates without downtime

Note: Most of the theoretical content above is covered in Kubernetes interactive tutorial .. I just formated it in a better way and removed most of the unnecessary details for the purpose of this topic ..

Demo

To make things easier for me :P .. I will use my google cloud dev machine which has Kubernetes installed and managing the cluster already "It has only one node".

You can easily install minikube to be able to play with the examples covered here on your local machine .. please refer to this article.

Step 1 - Configure Kube deployment/service:

Service.yaml and deployment.yaml for each service .. you can also include all of them into one template .. soon I will write about a very nice tool to template your configuration called Helm.

Deployment.yaml -- rest-service

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: rest-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rest-service
  template:
    metadata:
      name: rest-service
      labels:
        app: rest-service
    spec:
      containers:
        - name: rest-service
          image: docker.io/husseincoder/rest-service
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080

Service.yaml -- rest-service

apiVersion: v1
kind: Service
metadata:
  name: rest-service
spec:
  ports:
  - port: 8080
  selector:
    app: rest-service
  type: NodePort

Deployment.yaml -- grpc-service

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: grpc-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grpc-service
  template:
    metadata:
      name: grpc-service
      labels:
        app: grpc-service
    spec:
      containers:
        - name: grpc-service
          image: docker.io/husseincoder/grpc-service
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5000
            - containerPort: 5001

Service.yaml -- grpc-service

apiVersion: v1
kind: Service
metadata:
  name: grpc-service
spec:
  ports:
  - port: 5000
    name: passwords-endpoint
  - port: 5001
    name: health-endpoint
  selector:
    app: grpc-service
  type: NodePort

Step 2 - Create deployment and service:

se7so@se7so:~/rest-service/kube$ kubectl create -f .
deployment "rest-service" created
service "rest-service" created

se7so@se7so:~/grpc-service/kube$ kubectl create -f .
deployment "grpc-service" created
service "grpc-service" created

Lets test our deployments/PODS/Services that we have just created.

Step 3 - Show Deployments/PODS/Services:

se7so@se7so:~/$ kubectl get deployments
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
grpc-service   1         1         1            1           2m
heapster       1         1         1            1           10d
nginx          1         1         1            1           10d
rest-service   1         1         1            1           2m
se7so@se7so:~$ kubectl get pods
NAME                            READY     STATUS    RESTARTS   AGE
docker-gc-0hj9f                 1/1       Running   6          10d
grpc-service-1580463372-5hhwg   1/1       Running   0          2m
heapster-3746328914-mgwhr       1/1       Running   0          11h
ingress-lb-7s074                1/1       Running   6          10d
nginx-3110227365-44znv          1/1       Running   0          11h
rest-service-3837907542-kh3xt   1/1       Running   0          2m
se7so@se7so:~/$ kubectl get services
NAME           CLUSTER-IP   EXTERNAL-IP   PORT(S)                         AGE
graphite       None         <none>        2003/TCP                        10d
grpc-service   10.0.0.49    <nodes>       5000:32380/TCP,5001:32534/TCP   2m
kubernetes     10.0.0.1     <none>        443/TCP                         10d
logger         None         <none>        514/TCP                         10d
nginx          10.0.0.96    <nodes>       80:30001/TCP                    10d
proxy          None         <none>        3128/TCP                        10d
rest-service   10.0.0.225   <nodes>       8080:31326/TCP                  2m
se7so@se7so:~/IdeaProjects/dockerized-microservices/grpc-service/kube$

You can see I have other stuff running there .. but for now we are interested in those bold lines.

Notice in the service part .. for both grpc-service and rest-service the internal ports are mapped to external ones .. grpc-service ports 5000, 5001 are mapped to external ports 32380, and 32534 .. rest-service port 8080 is mapped to port 31326 which we shall use now to access our rest service.

Step 4 - Try it out:

se7so@se7so:~/$ curl http://10.132.0.117:31326/health && echo
{"status":"Running","dictSize":4758252}
se7so@se7so:~/$ curl http://10.132.0.117:31326/passwords?q=abc && echo
{"totalMatches":1268,"matches":["abcc","abccz","abcczyx","abccz911","abcczxy","abcczxy1","abccz247","abccymas","abccyes","abccyuki"]}

Step 5 - Scaling a deployment:

Lets scale our rest-service to have 2 replicas:

se7so@se7so:~/$ kubectl scale deployment rest-service --replicas=2
deployment "rest-service" scaled
se7so@se7so:~/IdeaProjects/dockerized-microservices/grpc-service/kube$ kubectl get deployments
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
grpc-service   1         1         1            1           10m
heapster       1         1         1            1           10d
nginx          1         1         1            1           10d
rest-service   2         2         2            2           10m
se7so@se7so:~/$ kubectl get pods
NAME                            READY     STATUS    RESTARTS   AGE
docker-gc-0hj9f                 1/1       Running   6          10d
grpc-service-1580463372-5hhwg   1/1       Running   0          10m
heapster-3746328914-mgwhr       1/1       Running   0          11h
ingress-lb-7s074                1/1       Running   6          10d
nginx-3110227365-44znv          1/1       Running   0          11h
rest-service-3837907542-kh3xt   1/1       Running   0          10m
rest-service-3837907542-rchms   1/1       Running   0          10s
se7so@se7so:~/IdeaProjects/dockerized-microservices/grpc-service/ku

As you can see in the first show deployments command .. the deployment is now has 2 replicas .. and each one is represented in a separate POD ..

The rest-service will take care of load balancing them and making sure the desired number of replicas will be running even if something wrong happens it will ask the master to recreate it.

Step 6 - check logs of a POD:

Since we have only one container so we can easily check the logs of the POD by issuing the following command.

se7so@se7so:~$ kubectl logs -f rest-service-3837907542-kh3xt

Step 7 - Describe deployment/pod/service:

In case you want see the details of one of your deployments/pods/services you can issue the following command:

se7so@se7so:~$ kubectl describe service rest-service
Name:   rest-service
Namespace:  default
Labels:   <none>
Selector:  app=rest-service
Type:   NodePort
IP:   10.0.0.225
Port:   <unset> 8080/TCP
NodePort:  <unset> 31326/TCP
Endpoints:  172.17.0.2:8080,172.17.0.8:8080
Session Affinity: None
No events.

se7so@se7so:~$ kubectl describe deployment rest-service
Name:   rest-service
Namespace:  default
CreationTimestamp: Fri, 28 Apr 2017 21:33:18 +0200
Labels:   app=rest-service
Selector:  app=rest-service
Replicas:  2 updated | 2 total | 2 available | 0 unavailable
StrategyType:  RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 1 max surge
Conditions:
  Type  Status Reason
  ----  ------ ------
  Available  True MinimumReplicasAvailable
OldReplicaSets: 
NewReplicaSet: rest-service-3837907542 (2/2 replicas created)
Events:
  FirstSeen LastSeen Count From    SubObjectPath Type  Reason   Message
  --------- -------- ----- ----    ------------- -------- ------   -------
  16m  16m  1 {deployment-controller }   Normal  ScalingReplicaSet Scaled up replica set rest-service-3837907542 to 1
  6m  6m  1 {deployment-controller }   Normal  ScalingReplicaSet Scaled up replica set rest-service-3837907542 to 2

Step 8 - Rolling an update:

Once you change your configuration to point to the new image or a new version of it .. you just issue the following command which is overrides what you have done in step 2 - without downtime ;).

se7so@se7so:~$ kubectl apply -f .

Step 9 - Open bash session on GRPC Service Container:

YES .. you can ssh to one of your containers and have full access to what's in there ..

se7so@se7so:~$ kubectl exec grpc-service-1580463372-5hhwg ls /home
app.jar
rockyou.txt
se7so@se7so:~$ kubectl exec -it grpc-service-1580463372-5hhwg bash
root@grpc-service-1580463372-5hhwg:/# ls /home
app.jar  rockyou.txt
root@grpc-service-1580463372-5hhwg:/# ps -a
  PID TTY          TIME CMD
  107 ?        00:00:00 ps
root@grpc-service-1580463372-5hhwg:/#

Super nice hah?! ;)

I think I have covered most of the important stuff .. but everyday I use this thing I discover more cool stuff .. please follow this series for more stuff like these.

Please contribute to this offer here.

Create/Manage docker swarm cluster

2017-04-18T14:32:00.001-07:00

Continuing the effort of previous post now I will show you how to create and manage a docker swarm cluster using docker-machine and docker-swarm

In docker swarm you create one or more managers and worker machines in the cluster .. the manager(s) take care of the orchestration of your deployed services (e.g., Creation/Replication/Assigning tasks to nodes/load balancing/service discovery ...).

Step 1: Create cluster machines using docker-machine

docker-machine create --driver virtualbox --virtualbox-memory "3000" master
docker-machine create --driver virtualbox --virtualbox-memory "3000" worker1
docker-machine create --driver virtualbox --virtualbox-memory "3000" worker2

In the above I've created 3 machines called (master, worker1, and worker2) with the same configuration and as I'll create the cluster on my local machine I've created them as virtual machines with docker engine installed on all of them. later we will initialize them as a swarm cluster.

Note: Docker machine can also allow you create these machines on digital ocean, AWS, and other third party cloud services .. they have other drivers .. please check docker-machine manual pages for more details.

You can also run commands on these machines without having to connect to them.

List all machines configured so far:

se7so@se7so:~$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER        ERRORS
master    *        virtualbox   Running   tcp://192.168.99.100:2376           v17.04.0-ce   
worker1   -        virtualbox   Running   tcp://192.168.99.102:2376           Unknown       Unable to query docker version: Get https://192.168.99.102:2376/v1.15/version: x509: certificate is valid for 192.168.99.101, not 192.168.99.102
worker2   -        virtualbox   Running   tcp://192.168.99.101:2376           Unknown       Unable to query docker version: Get https://192.168.99.101:2376/v1.15/version: x509: certificate is valid for 192.168.99.102, not 192.168.99.101

Show machine IP:

se7so@se7so:~$ docker-machine ip master
192.168.99.100
se7so@se7so:~$ docker-machine ip worker1
192.168.99.102
se7so@se7so:~$ docker-machine ip worker2
192.168.99.101

Step 2: Set up environment to run commands on master

eval "$(docker-machine env master)

Step 3: Init the swarm

You have to run this command on a manager machine.

docker swarm init --advertise-addr MASTER_MACHINE_IP

Note: This command will give you as output 2 commands one of them in order to join a machine as worker and the other one if you need to join a machine as a manager to the cluster swarm.

Step 4: Configure the worker machines to join the swarm

The below swarm join command is always generated after init swarm command above .. this will include the swarm cluster token you see below.

In order to run on both workers I will have to SSH to each one run the command then exit.

docker-machine ssh worker1
docker swarm join \
    --token SWMTKN-1-5ymh4597gc11bqq2keldy951fmwsqr4z8wjjcp47v5m43sv8qp-cbml91zj5xfilfi0syw1dl2o4 \
    192.168.99.100:2377
exit

docker-machine ssh worker2
docker swarm join \
    --token SWMTKN-1-5ymh4597gc11bqq2keldy951fmwsqr4z8wjjcp47v5m43sv8qp-cbml91zj5xfilfi0syw1dl2o4 \
    192.168.99.100:2377
exit

Step 5: Display all cluster nodes configured

se7so@se7so:~$ docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
govkclp1jeoy56e5kzwksd0z2 *  master    Ready   Active        Leader
lrl9lydrq7om79pp3aty1pea3    worker2   Ready   Active        
rlrt6252a1rjkzqdbig4a70bp    worker1   Ready   Active

Notice the master is configured as leader and the * is because we are currently connected to it.

Now I've configured a cluster with 1 manager and 2 workers .. lets run our rest-service and grpc-service from our previous post..

Step 6: Create a network to make our services visible to each other

docker network create -d overlay my_network

Step 7: Create services and publish its ports

For simplicity I've pushed both images to my docker hub account rest-service and grpc-service and now I will pull/run them on the cluster.

I've pushed them using the following commands:

docker tag rest-service husseincoder/rest-service
docker push husseincoder/rest-service
docker tag grpc-service husseincoder/grpc-service
docker push husseincoder/grpc-service

docker service create -p 80:8080 --name rest-service --network my_network husseincoder/rest-service 
docker service create -p 5000:5000 -p 5001:5001 --name grpc-service --network my_network husseincoder/grpc-service

Here I've just deployed the services with 1 replicas .. however I could have used --replicas parameter to set the number of replicas of the service.

You can also see that I've exposed the ports and also specified the network to make sure they can access each other using the service name

Step 8: Display all services

se7so@se7so:~$ docker service ls
ID            NAME          MODE        REPLICAS  IMAGE
ierr7y08cidg  rest-service  replicated  1/1       husseincoder/rest-service:latest
mhc7ffeqv01j  grpc-service  replicated  1/1       husseincoder/grpc-service:latest

This will display the services and show you many of the replicas has been started and how many are still being prepared .. it should show something like this:

Step 9: Display tasks of a service

se7so@se7so:~$ docker service ps rest-service
ID            NAME            IMAGE                             NODE     DESIRED STATE  CURRENT STATE           ERROR  PORTS
nc8m3zwjwivx  rest-service.2  husseincoder/rest-service:latest  worker2  Running        Running 13 minutes ago

In my case the manager decided to run it on the worker2 node .. it could be different in your case ;).

Step 10: Scale a service

As I've mentioned before I could have set the number of replicas of a service at creation time .. now since I've already started it lets scale one of them to run 5 replicas.

se7so@se7so:~$ docker service scale rest-service=3
rest-service scaled to 3

se7so@se7so:~$ docker service ls
ID            NAME          MODE        REPLICAS  IMAGE
ierr7y08cidg  rest-service  replicated  3/3       husseincoder/rest-service:latest
mhc7ffeqv01j  grpc-service  replicated  1/1       husseincoder/grpc-service:latest

se7so@se7so:~$ docker service ps rest-service
ID            NAME            IMAGE                             NODE     DESIRED STATE  CURRENT STATE           ERROR  PORTS
ezb2p3qo0tp2  rest-service.1  husseincoder/rest-service:latest  worker1  Running        Running 35 seconds ago         
nc8m3zwjwivx  rest-service.2  husseincoder/rest-service:latest  worker2  Running        Running 16 minutes ago         
uye5mlquy6l2  rest-service.3  husseincoder/rest-service:latest  master   Running        Running 35 seconds ago

Of course you can either scale up or down.

Step 11: Let's try calling the service

You can actually specify any of the cluster machine IPs not necessarily the manager.

Now you can start playing with docker-machine and docker-swarm and manage your own cluster .. Please share and go to the next topic - Kubernetes basics with demo.

Docker + Microservices all in one

2017-04-10T03:44:00.000-07:00

In the following topic(s) I will give a quick demo on how to use docker and docker-compose to streamline deployment of your application in dev/pre-prod environments with minimal effort.

This topic will continue for 2 to 3 articles till we reach to a point where we can CI/CD the whole application to production.

Keywords of the technologies used:

1. Spring Boot/REST

2. Google Protobuf/Grpc

3. Maven

4. Docker engine

5. Docker compose

6. Grafana + Graphite + StatsD

7. ElasticSearch + Logstash + Kibana (aka ELK)

The application basically consists of 2 microservices one of them (External) exposes 2 REST endpoints (JSON) and the other one (Internal) exposes 2 GRPC endpoints (ProtoBuf) .. the internal one is basically the backend and the external one is the client API ..

The application allows you to test complexity of a password by searching a dictionary of millions of passwords matches that share same prefix.

As part of the application set up there are few containers to monitor the application .. one of them has (Grafana+Graphite+stasD) images and the other one has (ElasticSearch + logstash + Kibana) for logs. (Not covered in this topic)

To get your hands dirty and try out the application yourself please clone the repository from here. (Feel free to contribute in case of issues or new features)

Lets quickly go through the configuration files and service implementation to give you an impression how the application works and services interact to each other.

protobuf-commons:

This module contains protocol buffer files definition .. if you are not familiar with protocol buffers please familiarize yourself with it here. To me its just enough to know that its the model classes definition for our backend service.

// model.proto           
syntax = "proto3";
package com.se7so.model;

option java_multiple_files = true;
option java_package = "com.se7so.model";
option java_outer_classname = "Model";

message FindPasswordsQuery {
    string query = 1;
}

message FindPasswordsResponse {
    int32 numOfMatches = 1;
    repeated string matches = 2;
}

message PasswordsServiceHealthStatus {
    string status = 1;
    int32 totalPasswordsLoaded = 2;
}

// services.proto

syntax = "proto3";
package com.se7so.services;

option java_multiple_files = true;
option java_package = "com.se7so.service";
option java_outer_classname = "Service";

import "google/protobuf/empty.proto";
import "model.proto";

service PasswordsService {
    rpc findPasswords (com.se7so.model.FindPasswordsQuery) returns (com.se7so.model.FindPasswordsResponse) {
    }
}

service PasswordsServiceHealthService {
    rpc getPasswordsServiceHealthStatus (google.protobuf.Empty) returns (com.se7so.model.PasswordsServiceHealthStatus) {
    }
}

grpc-service:

This module has the grpc endpoints implementation (PasswordsService and HealthService) that listen on ports 5000 and 5001.

The PasswordsService accepts FindPasswordsQuery and respond with a FindPasswordsResponse .. both are protobuf messages that are defined in the protobuf-commons .proto files.

The HealthStatusService doesn't have input but gives HealthServiceStatus response which contains information about the service (e.g., Running/Error and number of passwords loaded).

package com.se7so.grpc;

import com.se7so.dict.PasswordDictReader;
import com.se7so.model.FindPasswordsQuery;
import com.se7so.model.FindPasswordsResponse;
import com.se7so.service.PasswordsServiceGrpc;
import io.grpc.Server;
import io.grpc.ServerInterceptors;
import io.grpc.ServerServiceDefinition;
import io.grpc.stub.StreamObserver;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.Setter;
import lombok.extern.log4j.Log4j2;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;

import java.util.List;

@RequiredArgsConstructor(onConstructor =  @__(@Autowired))
@Log4j2
public class PasswordsService extends PasswordsServiceGrpc.PasswordsServiceImplBase implements GrpcService {

    @Getter
    @Value("${password.service.grpc.port}")
    private int port;
    @Value("${password.service.max.results}")
    private int maxResults;

    @Setter
    @Getter
    private Server server;
    private final GrpcServerInterceptor interceptor;
    private final PasswordDictReader reader;

    @Override
    public void findPasswords(FindPasswordsQuery request, StreamObserver responseObserver) {
        String prefix = request.getQuery();

        List results = reader.getDict().findPrefixes(prefix);
        int totalMatches = results.size();

        if(totalMatches > maxResults) {
            results = results.subList(0, maxResults);
        }

        responseObserver.onNext(FindPasswordsResponse.newBuilder()
                .addAllMatches(results)
                .setNumOfMatches(totalMatches)
                .build());

        responseObserver.onCompleted();
    }

    @Override
    public ServerServiceDefinition getServiceDefinition() {
        return ServerInterceptors.intercept(bindService(), interceptor);
    }
}

package com.se7so.grpc;

import com.google.protobuf.Empty;
import com.se7so.dict.PasswordDictReader;
import com.se7so.dict.PasswordTrie;
import com.se7so.model.FindPasswordsQuery;
import com.se7so.model.FindPasswordsResponse;
import com.se7so.model.PasswordsServiceHealthStatus;
import com.se7so.service.PasswordsServiceGrpc;
import com.se7so.service.PasswordsServiceHealthServiceGrpc;
import io.grpc.Server;
import io.grpc.ServerInterceptors;
import io.grpc.ServerServiceDefinition;
import io.grpc.netty.NettyServerBuilder;
import io.grpc.stub.StreamObserver;
import lombok.Data;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.Setter;
import lombok.extern.log4j.Log4j2;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;

import java.io.IOException;

@RequiredArgsConstructor(onConstructor =  @__(@Autowired))
@Log4j2
public class HealthStatusService extends PasswordsServiceHealthServiceGrpc.PasswordsServiceHealthServiceImplBase implements GrpcService {

    @Getter
    @Value("${health.status.grpc.port}")
    private int port;
    @Getter
    @Setter
    private Server server;
    private final GrpcServerInterceptor interceptor;
    private final PasswordDictReader passwordReader;

    @Override
    public void getPasswordsServiceHealthStatus(Empty request, StreamObserver responseObserver) {
        responseObserver.onNext(PasswordsServiceHealthStatus.newBuilder()
                .setStatus(passwordReader.getDict().size() == 0 ? "Error" : "Running")
                .setTotalPasswordsLoaded(passwordReader.getDict().size())
                .build());

        responseObserver.onCompleted();
    }

    @Override
    public ServerServiceDefinition getServiceDefinition() {
        return ServerInterceptors.intercept(bindService(), interceptor);
    }
}

The PasswordDictReader is basically our data store that loads passwords file and store it in memory as a Trie data structure .. it also has methods to search and get matches .. for simplicity you can check its implementation here.

rest-service:

In the rest service module there are REST endpoints that can be called easily from a browser to test the complexity of a password or to check the backend service health status.

Of course the rest-service communicate with the GRPC service by issuing a remote call to one of the services defined there .. and get back the response and map to DTO which is then given back as json to the client.


package com.se7so.rest;

import com.se7so.client.PasswordsServiceClient;
import com.se7so.model.PassServiceHealthDto;
import com.se7so.model.PasswordsResponseDto;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/")
public class RestServiceAPI {

    @Autowired
    private PasswordsServiceClient passwordsServiceClient;

    @RequestMapping(value = "/health", produces = "application/json")
    public PassServiceHealthDto getPasswordServiceHealthDto() {
        return passwordsServiceClient.getHealthStatus();
    }

    @RequestMapping(value = "/passwords", produces = "application/json")
    @ResponseBody
    public PasswordsResponseDto getPasswordsService(@RequestParam(value = "q") String query) {
        return passwordsServiceClient.findPasswordMatches(query);
    }
}

Docker:

In the grpc-service I use the openjdk:8 as the base image and copy the jar file of the grpc-service module to /home/app.jar also copying the dictionary of passwords to /home.

After copying all the needed files .. issuing a command to start the service.

# grpc-service

FROM openjdk:8

ADD target/grpc-service-1.0-SNAPSHOT.jar /home/app.jar
ADD rockyou.tar.gz /home

CMD java -jar /home/app.jar

The rest-service docker file is doing the same thing .. but no need for the dictionary file in this case ..

# rest-service

FROM openjdk:8

ADD target/rest-service-1.0-SNAPSHOT.jar /home/app.jar

CMD java -jar /home/app.jar

Docker Compose:

And here is the docker-compose file that defines how the services are defined and communicate together to be able to deploy all services and configurations using one command docker-compose up.

#docker-compose.yml

version: '2'
services:
  grpc-service:
    build: ./grpc-service/
    links:
     - graphite
    ports:
     - "5000:5000"
  rest-service:
    build: ./rest-service/
    links:
      - grpc-service
    ports:
      - "80:8080"
  graphite:
    image: "jlachowski/grafana-graphite-statsd"
    ports:
      - "2003:2003"
      - "8082:80"

The compose yml file contains definition of the three containers and exposes ports to the outside world .. also link the rest-service to the grpc-service to be able to call it at runtime.

Run:

Deploy the application using docker-compose:

cd /path/to/dockerized-microservices

mvn install # You will need this only once to build the application and generate binaries

docker-compose build # you will need this only once when you change the application

docker-compose up

Give it sometime to load services ...

 Go to http://localhost/health

Go to http://localhost/passwords?q=123
Go to http://localhost/passwords?q=mypass

Go to http://localhost:8082/dashboard/db/grpc-service-monitor

Clean up:

docker-compose stop # Stops the services without cleaning the containers

docker-compose down # Stops the services and clean up the created containers

Read the next topic in this series Create/Manage docker swarm cluster.

Lagrange multiplier to solve "Mushroom Scientists" problem

2017-01-02T05:56:00.000-08:00

Yesterday I was trying to solve this problem on codeforces and found its solution very interesting .. to be honest I had to do alot of research to understand the editorial solution .. so I managed to write a clearer documentation for how to solve these kind of problems.

First of all the problem is that you are trying to maximize f(x,y,z) = x^a Y^b Z^c with the constraint function g(x,y,z) = X+Y+Z = S ..

First of all if (a,b,c) = (0,0,0) then the maximum you get for f(x,y,z) is 1 :) .. otherwise lets solve the problem of at least having one of them > 0.

We can use Lagrange multiplier to solve this problem by doing the following:

=> f(x,y,z) = ƛg(x,y,z) -> Lagrange multiplier

=> x^a . y^b . z^c = ƛ(x+y+z)

by taking derivative d/dx we get:

=> ax^(a-1) . y^b . z^c = ƛ(1+0+0) = ƛ -> (1)

by taking the derivative d/dy we get:

=> bx^a . y^(b-1) . z^c = ƛ(0+1+0) = ƛ -> (2)

by taking the derivative d/dz we get:

=> cx^a . y^b . z^(c-1) = ƛ(0+0+1) = ƛ -> (3)

This implies the following:

=> ax^(a-1) . y^b . z^c = bx^a . y^(b-1) . z^c = cx^a . y^b . z^(c-1) = ƛ

=> (1) = (2) = (3)

By solving (1) and (2) together we get:

=> ax^(a-1) . y^b . z^c = bx^a . y^(b-1) . z^c = ƛ

=> ax^(a-1) . y^b = bx^a . y^(b-1)

=> bx = ay

=> a/x = b/y -> (4)

By solving (2) and (3) we get:

=> bx^a . y^(b-1) . z^c = cx^a . y^b . z^(c-1)

=> bz = cy

=> c/z = b/y -> (5)

From (4) and (5) we get:

a/x = b/y = c/z -> (6)

Now we can use this fact in g(x,y,z) we get:

=> x+y+z = S

=> x+bx/a+cx/a = S

=> ax+bx+cx = Sa

=> x = Sa/(a+b+c) -> (7)

Doing the same for y and z you get:

=> y = Sb/(a+b+c)

=> z = Sc/(a+b+c)

Now the problem is solved and here is my code:

Web Test Automation

2016-07-18T14:01:00.002-07:00

Most of tech companies now realise that manual testing is not as effective as anticipated and wanted to automate the whole process of users going to their websites performing some actions, submitting a form, or triggering a search,... etc.

You have probably thought many times of automating some repetitive tasks that you do every day or sometimes multiple times per day. Some people automate making their morning coffee - Believe me :).

Testing is one of the repetitive tasks that developers tend to do very often everyday .. Why? because it's important to make sure that things are working as expected before pushing them to production.

In this post I will not talk about the theory of automation or the frameworks themselves as there are plenty of articles for this purpose .. This article is just to introduce you to the world of automation and point you towards the next step.

Links to read

Test Automation

Selenium Framework

Phantom JS
Selenium headless mode

Steps to run this example

Create a maven project and copy these files
Make sure you download chrome driver from here
Edit the path to your chrome driver installation
Run the Example.java class

Example

In this example I will use Selenium framework to test Amazon DE website and search for something there .. then print the search page title .. No assertions/checks will be done as it depends on your business and the purpose of your test.

Pom.xml that defines libraries used:

Cheers ;)

Mockito for better unit tests

2016-03-10T13:40:00.000-08:00

Today I'm going to introduce you a very nice framework that allows you write good unit tests that cover only the parts of code that you want to test and stub all other dependencies .. but obviously in order to do that your code must be written in a testable way.

To know more about testability please refer to this wiki document.

Agenda

What is Mockito?

How to use it?

Example

What is Mockito?

Mockito is a framework that allows you writing unit tests that only test your piece of code by mocking and stub all other components that your piece of code depends on.

In other words if you want to test your integration with other libs and components then its not unit test any more .. its called integration test which is not covered in this post.

How to use it?

You can configure mockito using maven or simply download the lib from here.

For maven configuration you can define the dependency in your pom.xml as here.

Example

Note: I will use junit in the following example and in case you are not familiar with junit please refer to this.

Let's start by a very simple example that you have a class that reads a comma separated String and return it as a List<String> .. It reads this String by calling another class CSVReader that can either read it from file/database/network .. so in case you want to test your class only without being dependant on the CSVLineReaderClass you should mock the reader class.

Why?

Usually when running unit tests you need to get rid of the overhead of setting up other dependencies and want to see how your class/function behaves not how others behave.

Implementation of CSVLineParser:

The unit test should look like this, and you can add more test cases :).

When you run the test cases you will see the nice green results which means that all cases have passed correctly.

When you do something wrong like this :) .. you will see the unit tests failing.

In fact that's really nice, its really safe when you or someone else in your team mistakenly does these kind of stupid issues, its better to catch it during development than catching it in pre-production or on production and embrasse yourself and your team :D.

Mockito has way more than this simple example, but as usual I like to introduce you with very small nice things and then you can continue reading and see other features of the framework :).

I hope you like it.

Strong Connected Components

2015-10-18T14:00:00.000-07:00

There are plenty of problems that could be solved by finding the set of strongly connected components of a directed graph ..

In this topic I will explain what is a Strongly Connected Component? and how to find the set of all Strongly Connected Components of a directed graph.

Strong Connected Components

A Strongly connected graph is a graph where each vertex could be visited from each other vertex of the graph, for example the following graph has 5 strongly connected components.

Algorithm to find SCC

One of the most famous algorithms that is used to find strongly connected components of a graph is known by the Kosaraju's algorithm. And It works as follows:

DFS all nodes and save visited nodes in a stack S, push node only when finishing visit to all connected nodes.
Inverse the original graph by reversing all arcs E(U, V) to be E(V, U) instead
Pop each vertex V in S and DFS to get the strongly connected component that contains V.

Problem

To try this problem before reading my solution please visit CodeForces.

Your city has n junctions. There are m one-way roads between the junctions. As a mayor of the city, you have to ensure the security of all the junctions.

To ensure the security, you have to build some police checkposts. Checkposts can only be built in a junction. A checkpost at junction i can protect junction j if either i = j or the police patrol car can go to j from i and then come back to i.

Building checkposts costs some money. As some areas of the city are more expensive than others, building checkpost at some junctions might cost more money than other junctions.

You have to determine the minimum possible money needed to ensure the security of all the junctions. Also you have to find the number of ways to ensure the security in minimum price and in addition in minimum number of checkposts. Two ways are different if any of the junctions contains a checkpost in one of them and do not contain in the other.

Input

In the first line, you will be given an integer n, number of junctions (1 ≤ n ≤ 10⁵). In the next line, n space-separated integers will be given. The i^th integer is the cost of building checkpost at the i^th junction (costs will be non-negative and will not exceed 10⁹).

The next line will contain an integer m (0 ≤ m ≤ 3·10⁵). And each of the next m lines contains two integers u_i and v_i (1 ≤ u_i, v_i ≤ n; u ≠ v). A pair u_i, v_i means, that there is a one-way road which goes from u_i to v_i. There will not be more than one road between two nodes in the same direction.

Output

Print two integers separated by spaces. The first one is the minimum possible money needed to ensure the security of all the junctions. And the second one is the number of ways you can ensure the security modulo 1000000007 (10⁹ + 7).

Solution

Protocol Buffers

2015-09-21T11:33:00.000-07:00

In this topic I will talk about a very nice project that is done by Google which is Protocol Buffers...

What is Protocol Buffer?

Its a method of serializing structured data in an efficient, fast, and simple way where you can control how the object looks like using a predefined language.

So basically you define the object structure using a language which called "Interface description language" and usually its written in files with extension .proto and can be then compiled using protoc utility to generate Java, C++, Python class files, and to create instances you can use the builder class generated along with the generated classes.

Interface description language

A simple way to define the fields, types, and in case required or optional and you also have to assign sequence number for the tool to uniquely identify each of them when serializing and de-serializing. You can see the example below and read more about the language to develop more advanced object types.

I would say this is a good way to serialize and de-serialize objects to be totally independent of the language used especially in systems that involve communication between heterogeneous systems which may be developed with different technologies and/or languages. 

Complete Example

First you need to install protocol buffer compiler using the following command:

sudo apt-get install protobuf-compiler

Maven Build Script

Protocol Buffer File

Read and Write Proto Buff Objects (its just an example :D)

Monitoring tools and libraries

2015-09-18T13:06:00.001-07:00

This quarter I have been assigned a task to design and implement a solution to monitor the accuracy of our results and the health of our system (e.g., Request durations, Error rate, Caching ratio, .... etc).

In the beginning I felt it might be a silly task, but then it turned out being an exciting task that I've learnt alot from as there are many open source technologies I have used during this project.

In this topic I will mention some of these systems and libs that might be useful for many of you to track the health of your system and give you a good indication of how good your system is. ;)

Metrics Library

Simply if you want to monitor your system you should expose data to measure and correctly monitor your system. This library is used to expose some metrics and store them via JMX, however it supports other ways to expose and report your metrics (e.g., Console Reporter, JMX, HTTP, CSV, ....).

These metrics could be one of the following type:

Counters

You can use this metric to report something that increase and decrease over time (e.g., Bookings, Errors, Done requests, ... etc)

Histogram

I have found this metric type very useful to measure statistical changes of a sequence or series of data like (Request duration) .. for example you can update this metric with the duration of all done requests and measure at any point of time the mean, standard deviation, 75th percentile and so on to know how good your system at any point of time.

Timer

I didn't use this type of metrics but its mainly can be used for example to measure the duration of a request and get the rate of requests per second.

Health Checks

This one is also very useful if you have multiple subsystems that you need to check its health and see if anything happens, like health of database connection.

I used this lib and exposed all metrics via the JMX reporter and the results were amazing especially when you use other nice monitoring tools like the ones I will describe below.

Grafana

You can use Grafana to visualize your metrics and measure your system health over time.

I used Grafana to create multiple dashboards to measure the Error Rate, Cache Ratio, No Results Rate, and to measure the accuracy of our results.

I would recommend it for anybody wants to visualize and measure his system health and accuracy.

I will put some useful points about Grafana:

Datasource - You can define multiple data sources and each one has its own query editor which supported by Grafana (e.g., InfluxDB, Graphite, ...)
User - It supports user authentication and authorization via LDAP, Database, Google Authentication.
Dashboard - You can group graphs in one dashboard and create multiple dashboards to track your system.
Row - In one dashboard you can have multiple rows to organise how the graphs should look like.
Panel - Panel has multiple types but for me the most important was the graph which you can define your graph with the not very powerful query editor :D.

You can also define the period you need to see on each graph and the refresh rate of the whole dashboard.

Sample Images from Grafana website:

Non Technical Interview Tips

2015-05-09T18:04:00.002-07:00

After writing my previous topics that covers only technical parts of interviews, I received questions to cover the non technical skills needed for interviews. So I decided to talk only about this in a separate topic.

Rejection Reason Example

The reason was that I'm not good in communicating my ideas, and didn't prepare for the interview environment which was a bit messy.

I think the following points are really important to care about when you have an interview, feel free to send me other tips to include.

Interview Environment

Make sure that you set the best appointment for the interview which you are free and can sit in a quite place concentrating a solving very hard problems.
If you are not sure that the internet speed will be stable for a Skype interview, ask them to call you on your phone.
Make a small test for your microphone, sound, internet, ...

Listen

One of the important skill interviewers assess is that you listen to questions, understand them, and not approaching solutions before making sure that you understand the problem and corner cases.

Hints

If you can't find a solution or struggled in one, you can ask them for hints.

Answers

Try to give short answers, and avoid speaking too much.

Internet Connection

I prefer DSL than Wireless and 3G. Its more stable.

English

Try to speak slowly because they don't expect that you are super fluent in English.

Assumptions

If you have assumptions write them on the shared document, so that they feel you know what you are doing.

Feedback

Never ever ask for feedback within the interview.

HR vs Technical

Ask technical questions to tech people, and other questions like Salary, Vacations, Benefits, ... to HR.

Say No

If they asked you about something you don't know, feel free to say I don't know about this topic or technology. It would be ideal if you know something close to it and try to talk about instead.

Example

Do you know framework X?

No. But I know framework Y which does the same task, and I think the concept is the same, So you can ask me about the concept or I may answer according to my experience to Y.

Feel free to comment and share the topic :).

Interview Experience

2015-05-08T12:05:00.000-07:00

In this post I will talk about how to search, prepare, and pass a tough technical interview.

Searching for Jobs

I used to search for jobs but actually recruiters now are smart enough to find you if you have a good profile that could attract them calling you.

First make a good profile on Linkedin, and Xing. Put everything there and give small description about everything. In order to get noticed make sure that you include keywords that people use to search for candidates, and don't forget to update your current location and contact information regularly. Linkedin has a good job search that you can use to filter jobs that match your skills and needs e.g., Jobs by location.

Finally when a recruiter asks for your CV or contact, don't just reply with messages like "Hey please find the attached CV as you requested", prepare a template message like the following one:

Hello,

I'm a Software Engineer with solid algorithms, problem solving, and data structures skills. I have excellent experience in back-end development and you can check my skills and all the details about me in my CV and the following links and profiles.

- My blog contains lots of technical content (e.g., Algorithms, Problem solved by me, Administration, Java technology, Linux/Minix kernel topics, and some open source projects done by me) here is the link for it. Link

- My Github profile which contains all the projects done by me and projects I managed to contribute in. Link

- Topcoder profile: This website makes regular algorithms contests between the most professional engineers around the world and my progress in it is excellent. Link

- LinkedIn profile which contains general and detailed information about my experience both work experience and open source experience and it also contains most of the technologies I use and have knowledge about them. Link

- My SourceForge Page which contains binary packages of my projects for developers/students to download them and use them freely.

Preparation Time

Now you should put a plan for your preparation. I used to have the following plan to prepare for interviews:

Check Glassdoor for sample previous interviews of this company or a company with the same business.
Record all questions in one file with small categorization.
Spend only 1 day in this task and try to collect as much questions as you can.
Solve or review my solution at leetcode.
Checking my How to prepare for an interview series.
Studying problems in Cracking coding interview book.
Studying Hacking google interview handouts.
Solving 2 or 3 design questions.
Refresh my knowledge about Caching, Hashing, Distributed Systems, Parallel Computing, Computer Architecture, Unit Testing, Software Control Managers,...etc
Asking one of my friends to interview me.

Interview Process

Usually you will do a series of phone interviews 2 or 3, and you will be asked 2 or 3 coding questions that you can pass easily if you are well prepared. They may also ask you to write test cases for your solution and you have to cover all the cases.

For Testing:

Use the following strategy: As an example consider that he asks you to print an array of numbers:

Best cases test: {1, 2, 3}, {-1, -2, -3}, {1000000, 28, 27}
Load test {A very large array}, {INT_MAX, INT_MIN}
Random test {-1, 2, 77, 10082, 87273, 928, -91837, 28737}
Corner cases test {}, NULL, {1}, {-1}, {0}, {0, 1, -1}, {1, 2}

May be they need more than that and for such cases you can ask them for hint :) .. Its OK to ask for hints by the way ;)

At the end of the interview he may ask you some tricky questions, or open ended questions .. So be prepared.

Onsite Interview

In the onsite interview you will have around 5 or 6 technical interviews that will cover all aspects of skills that you may or may not have. Example:

3 Coding Interviews.
1 Architecture and Design Interview.
1 Interview with the Engineering Manager or Hiring Manager.

You should:

Be patient and listen to the full question.
Ask questions, usually they give you partial information to check what you will do!
Start demonstrating how would you approach this problem
When he asks you to code, start coding
Code carefully on the board, and try write clean code
When you finish coding, explain it briefly
He will ask you to analyse it, try to be smart and give him a proof for your analysis :).

Offer Time

Congratulations ;) you made it!
Reserve a Visa appointment
Read the contract carefully, and discuss it with the recruiter
Ask for relocation package
Negotiate your salary
Accept the offer
Prepare your papers
Good bye, don't come back :)

I hope you liked the post, please leave a comment and share it.

Project Lombok

2015-03-04T13:06:00.000-08:00

Lombok is a plugin that Java developers can use to save time and effort writing repeated code like one we see in data classes. I will describe it with examples and will include a link to video and the project site to download and try.

Consider the following data class Student which has some attributes and access to it Getters/Setters.

Too much code for a data class :-S

Look at Lombok and what it can do :-P

You can also control if you want only getters or setters using the @Getter and @Setter annotation before each attribute.

Now the code looks more pretty and readable as a data class. It has lots of other useful annotations that will save you lots of repeated code.

For example you can look at the @Cleanup annotation and what it does.

References:

FireMeUp Java Job Scheduler - Open Source

2014-12-01T03:42:00.000-08:00

In one of the projects at my current employer me and my team were asked to use a Java Job Scheduling library to schedule some jobs doing some business at dynamically configured time.

We decided to use Quartz as we were using Spring which provides a full integration to Quartz, and I was fascinated with the design and implementation of it.

I thought that I can start a new open source project that implements a Java Job Scheduler for learning purposes with some modifications/simplifications to the current design of Quartz library.

FireMeUp Scheduler is a simple Java scheduler that can be integrated within your application in a very simple way. The library is so simple and can be used in Small and Medium scale projects, and it provides the following functionality.

Jobs that run every interval of time.
Jobs that run every day, week, biweekly, monthly, bimonthly, and yearly in a specific time of the day. (e.g., Every day at 12:00 AM).
Having more that a scheduler in your application.
Enable/Disable scheduler freely throughout your application life time.
Configuring more than one trigger and associating them to schedulers.
Triggers can be configured in more than one way either a simple trigger that is scheduled to be triggered every interval of time, and crontab trigger that is triggered either daily, weekly, biweekly, monthly, bimonthly, or yearly at specific time HH/MM/SS.
Implementing your jobs and associating them to triggers, more than one job to each trigger.
Register/Unregister triggers to schedulers.
Register/Unregister jobs to triggers.
Job can't overlap itself if its in a running state.
You can implement your own trigger by extending the AbstractTrigger that contains most of the business.

Future Functions

Here are some ideas I need to add to the current project, and anyone who needs to start his own open source life can start implementing them and be part of the project.

Adding state recovery functionality for the configured Schedulers, Triggers, Jobs, I think we can use in-memory (not recovered) and database (recovered) models.
Threading management functionality, manage the number of concurrent running threads.
Configuration management functionality, configuring the application programmatically and using a properties file.
Providing a crontab expression parser functionality.
Documentation of the whole project.
More complex examples and test cases.

Project Links