tag:blogger.com,1999:blog-5138930151916967872023-03-07T05:43:21.817+05:30On Code, And Other ThingsUnknownnoreply@blogger.comBlogger35125tag:blogger.com,1999:blog-513893015191696787.post-16369701552587976332020-02-01T00:14:00.000+05:302020-02-01T00:14:46.064+05:30Automating Heap Dumps For Java Containers in Google Kubernetes Engine<div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Heap dumps are an indispensable tool for debugging memory issues in Java processes. The typical way of taking a memory dump is using the jmap command</span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></div><div style="text-align: justify;"><span style="color: blue;"><b><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;">jmap -dump:format=b;file=/tmp/heap.dump 2592</span></span></b></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">This will trigger a heap dump for the process with id 2592 (assuming it's a Java process) and store it in a file /tmp/heap.dump. This can be analyzed later with a heap dump analyzer like <a href="https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jhat.html" target="_blank">jhat</a>, <a href="https://www.eclipse.org/mat/" target="_blank">MAT</a>, or <a href="https://visualvm.github.io/" target="_blank">VisualVM</a> - which are free tools.</span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Triggering a heap dump for a Java process that is running in a container, inside a pod, in a Google Kubernetes Engine (GKE) cluster node is not so straightforward. There are many layers of infrastructure that you have to cross to get at the Java process. Your process would usually run as part of a managed abstraction like a Deployment or a StatefulSet in your Kubernetes cluster. Your starting point would be just the pod name.</span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">But,</span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">- Knowing the pod name is not enough - you also have to locate the cluster node where it's running and ssh into it.</span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">- A GKE node might be running many pods, and many Java processes - you have to identify the correct one once once you have ssh'ed into it. "docker ps" can help here.</span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">- The GKE node might not have jmap. It's not straightforward to install the JDK there because it would typically be running <a href="https://cloud.google.com/container-optimized-os/docs/" target="_blank">COS</a>. So you have to get inside the container and trigger the dump.</span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">- You have to copy the dump to an accessible location, maybe a GCS bucket, from where you can download it to analyze. Uploading to a GCS bucket requires gsutil, which is not present by default in a COS node. </span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">I have automated this entire process using just shell scripts and gcloud commands. The source code <a href="https://github.com/talonx/gke-container-debug" target="_blank">is on GitHub</a>. They also use the toolbox utility that Google provides as a container for running debug tools. Invoking "toolbox" inside your GKE node will launch this container.</span></span><br /><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">These scripts have an assumption which might not be valid for your cluster - I'll point it out at the relevant point in the code.</span></span></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Here's a step by step explanation of the flow.</span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">There are 3 shell scripts - <a href="https://github.com/talonx/gke-container-debug/blob/master/src/k8s-debug-client.sh">k8s-debug-client.sh</a> being the one to run from your dev box or bastion host. This one invokes k8s-debug-vm.sh (inside the GKE cluster node, i.e. the VM) which in turn invokes the k8s-debug-toolbox.sh.</span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">First we find out the node on which the pod is running</span></span><br /><br /><b><span style="color: blue;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> <span style="font-family: "courier new" , "courier" , monospace;">node_name=<span class="pl-s"><span class="pl-pds">`</span>kubectl get pod <span class="pl-smi">${pod_name}</span> -o json <span class="pl-k">|</span> jq <span class="pl-s"><span class="pl-pds">'</span>. | .spec.nodeName<span class="pl-pds">'</span></span><span class="pl-pds">`</span></span></span> </span></span></span></b><br /><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">and get its public IP</span></span><br /><br /><span style="color: blue;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace;"><b>public_ip=<span class="pl-s"><span class="pl-pds">`</span>gcloud compute instances list --filter=<span class="pl-s"><span class="pl-pds">"</span>name=(<span class="pl-smi">${node_name}</span>)<span class="pl-pds">"</span></span> --format=<span class="pl-s"><span class="pl-pds">"</span>value(networkInterfaces[].accessConfigs[0].natIP)<span class="pl-pds">"</span></span><span class="pl-pds">`</span></span></b></span> </span></span></span><br /><br /><span style="font-size: small;"><span style="font-family: "arial" , "helemvetica" , sans-serif;">copy the other two scripts to it</span></span><br /><br /><span style="font-family: "Courier New", Courier, monospace;"><span style="color: blue;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> <b>scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i <span class="pl-smi">${keyfile}</span> k8s-debug-vm.sh k8s-debug-toolbox.sh <span class="pl-smi">${user}</span>@<span class="pl-smi">${public_ip}</span>:</b> </span></span></span></span></span><br /><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">and trigger them</span></span></div><div style="text-align: justify;"><br /><span style="color: blue;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> <b>ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i <span class="pl-smi">${keyfile}</span> <span class="pl-smi">${user}</span>@<span class="pl-smi">${public_ip}</span> sh k8s-debug-vm.sh <span class="pl-smi">${pod_name}</span> <span class="pl-smi">${action}</span> <span class="pl-smi">${bucket}</span></b> </span></span></span></span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"></span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">I turn off the ssh warnings as I trigger this from a CI system on demand. If you run them manually, you can remove the -o options. </span></span><br /><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Inside the GKE node, <a href="https://github.com/talonx/gke-container-debug/blob/master/src/k8s-debug-vm.sh">k8s-debug-vm.sh</a> figures out the correct container id and uses docker exec to trigger a heap dump inside it. </span></span><br /><br /><span style="font-family: "Courier New", Courier, monospace;"><span style="color: blue;"><span style="font-family: "Courier New", Courier, monospace;"><b><span style="font-family: "Courier New", Courier, monospace;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">container_id=<span class="pl-s"><span class="pl-pds">`</span>docker ps <span class="pl-k">|</span> grep <span class="pl-smi">${pod_name}</span> <span class="pl-k">|</span> grep -v <span class="pl-s"><span class="pl-pds">"</span>POD<span class="pl-pds">"</span></span> <span class="pl-k">|</span> awk <span class="pl-s"><span class="pl-pds">'</span>{print $1}<span class="pl-pds">'</span></span><span class="pl-pds">`</span></span></span></span></span></b></span></span></span><br /><span style="font-family: "Courier New", Courier, monospace;"><span style="color: blue;"><span style="font-family: "Courier New", Courier, monospace;"><b><span style="font-family: "Courier New", Courier, monospace;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="pl-s"><span class="pl-pds">docker <span class="pl-c1">exec</span> <span class="pl-smi">${container_id}</span> sh -c <span class="pl-s"><span class="pl-pds">"</span>jmap -dump:format=b,file=heap.dump 1<span class="pl-pds">"</span></span> </span></span></span></span></span></b></span></span></span><br /><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Note that the heap dump is inside the container, and not in the VM. You may not be able to push it to a GCS bucket from the container as there is no gsutil and no permissions. So we need to copy the dump to the VM. We copy it to a uniquely named file.</span></span><br /><br /><span style="color: blue;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: "Courier New", Courier, monospace;"><b><span style="font-family: "Courier New", Courier, monospace;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">dttime=<span class="pl-s"><span class="pl-pds">`</span>echo <span class="pl-s"><span class="pl-pds">$(</span>date <span class="pl-s"><span class="pl-pds">'</span>+%d-%b-%Y-%H-%M-%S<span class="pl-pds">'</span></span><span class="pl-pds">)</span></span><span class="pl-pds">`</span></span></span></span></span></b></span></span></span><br /><span style="color: blue;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: "Courier New", Courier, monospace;"><b><span style="font-family: "Courier New", Courier, monospace;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="pl-s"><span class="pl-pds">filename=<span class="pl-smi">${pod_name}</span>-<span class="pl-smi">${dttime}</span>.hdump</span></span></span></span></span></b></span></span></span><br /><span style="color: blue;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: "Courier New", Courier, monospace;"><b><span style="font-family: "Courier New", Courier, monospace;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span class="pl-s"><span class="pl-pds">docker cp <span class="pl-smi">${container_id}</span>:heap.dump <span class="pl-smi">${filename}</span> </span></span></span></span></span></b></span></span></span><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Now you have the dump file in the VM but there is no gsutil. So you need to invoke toolbox, which has gsutil inside it. Does that mean we need to copy the dump inside toolbox now? No, because toolbox mounts several useful directories by default from the VM it's running on. </span></span><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">So we just invoke toolbox and pass it the path to the </span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><a href="https://github.com/talonx/gke-container-debug/blob/master/src/k8s-debug-toolbox.sh">k8s-debug-toolbox.sh</a> (which is in the home directory of the user you are logged in as in the VM) as it would appear from inside toolbox (since the home directory is also mounted inside toolbox).</span></span><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Inside toolbox, we can use gsutil to upload the dump file (which is also available inside toolbox because it's in the home directory of the user you are logged in as in the VM). But here's a catch. gsutil requires permissions to upload to a GCS bucket. One way to provide this permission is with an IAM permissions JSON file. But how does it get to the VM? </span></span><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">This is the caveat I mentioned above. In the infrastructure I manage, almost every Java pod has a config map with a permissions file that the pod uses to access Google Cloud services. This file is accessible as a mounted directory inside toolbox, so, voila! </span></span><br /><br /><span style="color: blue;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: "Courier New", Courier, monospace;"><b><span style="font-family: "Courier New", Courier, monospace;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">/google-cloud-sdk/bin/gcloud auth activate-service-account --key-file=<span class="pl-smi">${dir}</span>/key.json</span></span></span></b></span></span></span><br /><span style="color: blue;"><span style="font-family: "Courier New", Courier, monospace;"><span style="font-family: "Courier New", Courier, monospace;"><b><span style="font-family: "Courier New", Courier, monospace;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">/google-cloud-sdk/bin/gsutil cp /media/root/home/<span class="pl-smi">${user}</span>/<span class="pl-smi">${filename}</span> gs://<span class="pl-smi">${bucket}</span>/kdev-debug/<span class="pl-smi">${filename}</span></span></span></span></b></span></span></span><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">If you don't have this shortcut, you will a need to set the permissions somehow. There are multiple ways of doing it - one being to run a custom container instead of toolbox that has gsutil installed and can mount a config map which has the permissions. Another is to upload the permissions file to the VM when you run the command, use it from inside toolbox, and then delete it. The second one is a tad risky.</span></span><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">These scripts can be modified to be usable for any Kubernetes cluster and not just GKE. Most of the changes will be in the commands that fetch the list of running nodes. If you are using another OS for your K8S VMs, you can install Java directly on the VM and trigger the dump, after you find out the mapping between the container ids and the process ids as visible from the VM.</span></span></div><div style="text-align: justify;"><br /></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-41803221085794378662019-11-28T22:15:00.000+05:302019-12-08T18:23:35.580+05:30The K8S Networking Implementation in Google Kubernetes Engine<span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">I was recently digging into some finer points of exposing Kubernetes pods as services and came across <a href="https://www.youtube.com/watch?v=y2bhV81MfKQ&feature=youtu.be" target="_blank">this fantastic talk</a> from Google Cloud Next '17. It's about how Kubernetes networking works on the Google Cloud Platform.</span></span><br /><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">The Kubernetes specification dictates, among other things, the networking requirements for deployment. On the Google Cloud Platform (GCP), K8S is available as the Google Kubernetes Engine (GKE) product. GKE is part of GCP, and uses Google compute instances as the K8S hosts and its virtual network for network traffic. </span></span></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Why is the networking a big deal? Is the specification not already implemented by the K8S project? It is - however, it needs an underlying set of compute, storage and network resources to function. These resources are usually provided by a cloud provider, or bare metal machines + cloud management software, if you are hosting your own cloud. A cloud provider has to go some extra distance to ensure it meets the K8S spec requirements - because it's providing a virtualized environment, and not all things might work as it does in a non-virtualized one. </span></span><br /><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">The talk is about how Google does it for GKE.</span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">I've summarized some of the interesting points, leaving out the vanilla Kubernetes details which are easily found in the documentation.</span></span></div><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span><br /><h2><span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Internal Traffic </span></span></h2><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Linux network namespaces and virtual interfaces are used as the foundation. </span></span><br /><br /><h4><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">For two pods to talk to each other</span></span></h4><ul><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Each VM (K8S cluster host) has a root network namespace (usually eth0)</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Each pod in that host has its own network namespace, separate from the root</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">For these to talk to each other, we use a pipe between two virtual interfaces, one end of which shows up as (again, usually) vethxx in the VM, and the other end as eth0 in the pod.</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">For two pods to talk to each other, we need a bridge between the vethxxs in the VM, which is (usually) named cbr0. This uses ARP to determine where to route packets.</span></span></li></ul><h4><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">For two pods to talk to each other <i>across</i> VMs</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span></h4><ul><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">The network between VMs has to know how to route packets whose src and dest are both pods.</span></span> </span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Each VM has an IP block from which it allocates IPs to pods inside it.</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"></span><span style="font-family: "arial" , "helvetica" , sans-serif;">Once the packet leaves a pod and reaches the bridge, it gets sent out the default route as there is entry on that VM's ARP table for that dest pod IP.</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span><span style="font-family: "arial" , "helvetica" , sans-serif;">At this point, the packet will be dropped by GCP's network as the source IP does not match the VM's IP ("anti-spoof"). To get around this, each VM is setup to be able to forward packets, and disable the anti-spoof mechanism. One static route for each VM is setup on the network to route packets for that VM's pod IP range.</span></span></li></ul><h4><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">To route pod packets to a pod behind a Service</span></span></h4><ul><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Once the packet hits the bridge, it's processed by an iptables rule.</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">iptables first chooses a pod for the Service, load balancing between different pods. In iptables proxy mode, it <a href="https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-iptables" target="_blank">chooses backends randomly</a>.</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">iptables then performs a DNAT, changing the destination IP in the packet to that of the dest pod. There is a tool called conntrack that keeps track of the fact that a connection was made to the pod's IP for a packet meant for the Service IP. </span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">The packet is routed as usual from src pod to dest pod</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">iptables rewrites the src IP to the Service IP in the response packet before sending it to the pod which made the request</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">iptables is, in general, routing traffic to pods behind a Service.</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">kube-proxy just configures and syncs iptables rules based on changes fetched from the K8S API - the name does not reflect anything about its function. It's a legacy name.</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">DNS runs as a Service, in a pod, in K8S.</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span></li><ul><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Special needs - particular Service IP, autoscaled to the cluster size.</span></span></li></ul></ul><h3><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span></h3><h2><span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif;">External Traffic </span></span></h2><h3><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"></span></span></h3><h4><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span></h4><h4><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">From a pod to the internet</span></span></h4><ul><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">A packet's internal address is rewritten to the external IP of the VM on which the pod is running, so that the internet knows where it came from. The reverse rewrite happens on the way back.</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Before the traffic goes out of the VM, iptables rewrites the pod's src IP to the VM's internal IP. After this, the same thing happens as in the previous point.</span></span></li></ul><h4><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">From the internet to a pod using Service type: LoadBalancer</span></span></h4><ul><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Service type: LoadBalancer creates a network LB in GCP, pointing the GCP forwarding rule to all the VMs in the K8S cluster</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Google's NLB is a packet forwarder, not a proxy, making it possible to read the original client's IP address from the packet directly. In the L7 ingress LB, this is achieved by the X-Forwarded-For header.</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">LB chooses a VM, which may or may not have the pod (or any pods for that matter) the packet is meant for.</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">iptables on the VM chooses a pod. If it's on a different VM, a DNAT happens like before changing the dest to the pod's IP, instead of the LB's IP.</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">There is a second NAT happening here, changing the src from the client, to this VM's IP. This ensures the original VM on which the packet lands stays in the flow. If this does not happen, and the packet is sent to a different VM from this one, and the response goes back to the NAT layer just before the LB, it will be dropped since the packet was sent to the first VM, and not this one, or the pod where it ended up. This loses the original client IP information though.</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Once it lands on the other VM, it gets routed to the pod, and the response goes back, with all the reverse NAT happening on the way back.</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">The "imbalance" here that can be caused by the LB knowing only about VMs, and not about pods, is mitigated by re-balancing inside K8S between pods. This balancing is random and apparently is "well-balanced" in practice, but can cause an extra network hop, and the client IP is hidden from the pod.</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">There is an annotation to tune this part.</span></span></li></ul><br /><script src="https://gist.github.com/talonx/d03959515793f9970aa76e79efefd9c9.js"></script> Note that this annotation has been superseded by another property since this talk. <br /><script src="https://gist.github.com/talonx/72fd77618a0c72eb1b84f6a7020e3498.js"></script> <span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Setting this will lead to iptables always choosing a pod on the same node, which also preserves the client IP, but risks imbalance.</span></span><br /><br /><h4><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">From the internet to a pod using an Ingress LoadBalancer</span></span></h4><ul><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">The NodePort service port forwards to the pod(s) using iptables, like before</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Source IP of a packet is the internal address of the LB, not the external one. This one is a proxy.</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">The SNAT/DNAT works as in the previous case</span></span></li><li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">To avoid the extra network hop, the same OnlyLocal annotation works. </span></span></li></ul><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">This talk is more than 2 years old. Since then, there have been newer developments in GKE, including "<a href="https://cloud.google.com/kubernetes-engine/docs/how-to/container-native-load-balancing" target="_blank">container-native load balancing</a>" and in K8S itself, e.g., <a href="https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/" target="_blank">IPVS based load balancing</a>.</span></span><br /><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-17436257484198551632019-01-06T20:50:00.001+05:302023-03-06T20:59:32.570+05:30Automagically Discovering and Scraping Google Compute Nodes in PrometheusPrometheus can scrape metrics from either a static list of machines or discover machines dynamically using a service discovery plugin. Service discovery plugins exists for the major cloud providers, which includes Google Cloud Platform (GCP). <br /><br />A simple configuration for GCP’s service discovery in the Prometheus config (usually prometheus.yml) looks like this<br /><pre><code><br /> - job_name: node<br /><br /> honor_labels: true<br /><br /> gce_sd_configs:<br /><br /> - project: ml-platform-a<br /><br /> zone: us-eastl1-a<br /><br /> port: 9100<br /><br /> relabel_configs:<br /><br /> - source_labels: [__meta_gce_label_cloud_provider]<br /><br /> target_label: cloud_provider<br /><br /> - source_labels: [__meta_gce_label_cloud_zone]<br /><br /> target_label: cloud_zone<br /><br /> - source_labels: [__meta_gce_label_cloud_tier]<br /><br /> target_label: cloud_tier<br /><br /> - source_labels: [__meta_gce_label_cloud_service]<br /><br /> target_label: cloud_service<br /><br /> - source_labels: [__meta_gce_instance_name]<br /><br /> target_label: instance<br /></code></pre>Let’s dissect this. Running Prometheus with this configuration will fetch all the instances in the GCP project ml-platform-a in the zone us-east1-a, and scrape their "/metrics" endpoints at port 9100. The relabel config lets you convert GCE (Google Compute Engine) labels (<i>source</i>) into Prometheus labels (<i>target</i>). <br /><br />However, this config will attempt to pull data from all instances whether they are running or not, and end up marking the stopped ones as "DOWN". To get around this, you need to filter out the stopped instances. Add a filter after the port directive, like this<br /><blockquote class="tr_bq"> port: 9100<br /> filter: '(status="RUNNING")'</blockquote>The equivalent gcloud command to list all running instances looks like <br /><blockquote class="tr_bq">gcloud compute instances list --filter='status:(RUNNING)</blockquote>Note the difference in syntax. The keywords, however, are identical.<br /><br />What if you have multiple exporters running on a specific set of instances? You can select them by their label(s) and add a different gce_sd_config section for them. For instances which have exporters running on say, port 3000, and have a label called “cloud_service:dashboard”, the config would look like<br /><pre><code><br /> - job_name: dashboard<br /><br /> honor_labels: true<br /><br /> gce_sd_configs:<br /><br /> - project: ml-plaform-a<br /><br /> zone: us-central1-c<br /><br /> port: 3000<br /><br /> filter: '(status="RUNNING") AND (labels.cloud_service="dashboard")'<br /><br /> relabel_configs:<br /><br /> - source_labels: [__meta_gce_label_cloud_provider]<br /><br /> target_label: cloud_provider<br /><br /> - source_labels: [__meta_gce_label_cloud_zone]<br /><br /> target_label: cloud_zone<br /><br /> - source_labels: [__meta_gce_label_cloud_tier]<br /><br /> target_label: cloud_tier<br /><br /> - source_labels: [__meta_gce_label_cloud_service]<br /><br /> target_label: cloud_service<br /><br /> - source_labels: [__meta_gce_instance_name]<br /><br /> target_label: instance<br /></code></pre>Just for reference, the analogous gcloud command is<br /><blockquote class="tr_bq">gcloud compute instances list --filter='status:(RUNNING) AND labels.cloud_service:dashboard'</blockquote>The relabel_configs is identical to that of the 9100 scraper. It would have been nice if Prometheus had allowed for a common relabel config section that could be reused for such cases.<br /><br />The GCE service discovery plugin needs read permission on the GCE Compute API to be able to pull the list of instances. There are several ways to do this, depending on how you are running Prometheus<br /><br /><ul><li>Prometheus on a GCE instance in the same project : You can assign the correct IAM permissions to your GCE instance, and nothing more needs to be done. </li><li>Prometheus on a GCE instance in a different project, or a non-GCE machine : You can create a service account in your GCP project, download the key as a JSON and start Prometheus with the JSON set in an environment variable, like this </li></ul><blockquote class="tr_bq">GOOGLE_APPLICATION_CREDENTIALS=...path..to..json..credentials… ./prometheus -- (other options)</blockquote>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-27608929020363039952016-05-14T21:52:00.003+05:302016-05-14T21:53:19.474+05:30Executing External Commands in Go<span style="font-family: Arial, Helvetica, sans-serif;">Sometimes we need to invoke operating system commands from our code. Most languages have APIs for this - Java has Runtime.exec(), Python has subprocess and Go has the <a href="https://golang.org/pkg/os/exec/" target="_blank">os/exec</a> package. This post briefly explores the Go API.</span><br /><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;">The APIs are part of the exec/os package. The <a href="https://golang.org/pkg/os/exec/#Cmd" target="_blank">Cmd</a> abstraction encapsulates a command object, where various tweaks can be done including setting the standard output and error streams.</span><br /><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;">Simple execution of a command is very easy. However, if one wants finer control over the execution, including control over streams and the correct exit code, maybe when it's to be used in a framework or a library, the code becomes slightly more involved. </span><br /><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;">Creating the Cmd object is straighforward</span><br /><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Courier New, Courier, monospace;"> cmd := exec.Command(binaryName, args...) </span><br /><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;">The output and error streams can be redirected as follows</span><br /><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Courier New, Courier, monospace;"> stdout := &bytes.Buffer {}</span><br /><span style="font-family: Courier New, Courier, monospace;"> stderr := &bytes.Buffer {}</span><br /><span style="font-family: Courier New, Courier, monospace;"> cmd.Stdout = stdout</span><br /><span style="font-family: 'Courier New', Courier, monospace;"> cmd.Stderr = stderr</span><br /><div style="-webkit-text-stroke-width: 0px; color: black; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px;"><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div><div style="-webkit-text-stroke-width: 0px; color: black; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px;"><span style="font-family: Arial, Helvetica, sans-serif;">Once the command has been executed, it returns an Error object if the execution failed.</span></div><div style="-webkit-text-stroke-width: 0px; color: black; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px;"><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Courier New, Courier, monospace;"> err := cmd.Run()</span></div><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;">The command execution can fail for various reasons - it might not have been a valid command, it might have exited with an error code or their might have been IO errors. We need to detect these cases so that the caller of the API gets the correct response.</span></div><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;">The Go source file <a href="https://golang.org/src/os/exec/exec.go" target="_blank">exec.go</a> documents the error types that can occur.</span></div><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div><h4 style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;">exec.ExitError</span></h4><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;">An unsuccessful exit by a command. The ExitError object also has a "<span class="comment">subset of the standard error output from the C</span><span class="comment">md.Output method if standard error was not otherwise being </span><span class="comment">collected." <quote docs>.</span></span></div><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div><h4 style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;">exec.Error</span></h4><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;">One of the cases where this Error can be returned is when the command could not be located. When the Command struct instance is created, it calls the LookPath method to locate the binary if the binaryName argument does not have path separators, which can return one of these Error instances when the executable could not be located. The actual implementation <a href="https://golang.org/src/os/exec/lp_windows.go" target="_blank">depends</a> <a href="https://golang.org/src/os/exec/lp_unix.go" target="_blank">on the</a> OS.</span></div><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;">We can switch on the Error type</span></div><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Courier New, Courier, monospace;"> switch err.(type) {</span><br /><span style="font-family: Courier New, Courier, monospace;"> case *exec.ExitError:</span><br /><span style="font-family: Courier New, Courier, monospace;"> e := err.(*exec.ExitError)</span><br /><span style="font-family: Courier New, Courier, monospace;"> if status, ok := e.Sys().(syscall.WaitStatus); ok {</span><br /><span style="font-family: Courier New, Courier, monospace;"> exitcode = status.ExitStatus()</span><br /><span style="font-family: Courier New, Courier, monospace;"> }</span><br /><span style="font-family: Courier New, Courier, monospace;"> case *exec.Error:</span><br /><span style="font-family: Courier New, Courier, monospace;"> e := err.(*exec.Error)</span><br /><span style="font-family: Courier New, Courier, monospace;"> stderr.WriteString(e.Err.Error())</span><br /><span style="font-family: Courier New, Courier, monospace;"> default:</span><br /><span style="font-family: Courier New, Courier, monospace;"> panic("Unknown err type: " + reflect.TypeOf(err).String())</span><br /><span style="font-family: Courier New, Courier, monospace;"></span><br /><span style="font-family: Courier New, Courier, monospace;"> }</span><br /><div><br /></div><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;">If it's ExitError, we need to query the OS specific implementations using the <a href="https://golang.org/pkg/os/#ProcessState.Sys" target="_blank">Sys interface</a>. The Unix implementation is syscall.WaitStatus. </span></div><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;">if the err instance is nil, the command execution succeeded and we can get the exit code from the Cmd itself.</span></div><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Courier New, Courier, monospace;"> if status, ok := cmd.ProcessState.Sys().(syscall.WaitStatus); ok {</span><br /><span style="font-family: Courier New, Courier, monospace;"> exitcode = status.ExitStatus()</span><br /><span style="font-family: Courier New, Courier, monospace;"></span><br /><span style="font-family: Courier New, Courier, monospace;"> }</span><br /><div><br /></div><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif;">The complete source code is <a href="https://github.com/talonx/go-sys-utils/blob/master/executecommand.go" target="_blank">here</a>. </span></div><div style="margin: 0px; orphans: auto; text-align: start; text-indent: 0px; widows: 1;"><span style="font-family: Arial, Helvetica, sans-serif; font-size: x-small;"><br /></span></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-41669400376341420102015-06-03T18:53:00.002+05:302015-06-03T18:54:30.349+05:30Principles of Reactive Programming - Coursera MOOC - Review<span style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;">The recently concluded </span><a href="https://www.coursera.org/course/reactive" style="font-family: arial, sans-serif; font-size: 12.8000001907349px;" target="_blank">Principles of Reactive Programming</a><span style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;"> on Coursera was a good introduction to the paradigm of Reactive Programming in the Scala programming language. It was a kind of sequel to "</span><a href="https://www.coursera.org/course/progfun" style="font-family: arial, sans-serif; font-size: 12.8000001907349px;" target="_blank">Functional Programming Principles in Scala</a><span style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;">" from last year. I say kind of as you can still take this course without taking the first one provided you have familiarity with Scala and functional programming ideas.</span><br /><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;"><br /></div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;">In a nutshell, here is what I think about the course.</div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;"><br /></div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;">It's an introduction to a different mode of concurrent programming, to <a href="http://www.reactivemanifesto.org/" target="_blank">reactive principles</a>, all using Scala libraries. It does not go into much depth (which is probably a drawback of most MOOCs) but provides a foundation on which one can build. For example, I can dive deeper into Actor programming now that I know the fundamentals.</div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;"><br /></div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;"><b>Pros</b></div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;">- Great introduction to Reactive Programming</div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;">- Instructors are experts in their fields (<a href="http://en.wikipedia.org/wiki/Martin_Odersky" target="_blank">Martin Odersky</a>, <a href="http://en.wikipedia.org/wiki/Erik_Meijer_%28computer_scientist%29" target="_blank">Eric Meijer</a>, <a href="http://rolandkuhn.com/" target="_blank">Roland Kuhn</a>)</div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;">- Assignments corresponding to every week's topic</div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;"><br /></div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;"><b>Cons</b></div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;">- Differences in teaching styles and video content among the three instructors make the ride jumpy. Or maybe I am just spoilt after taking Martin Odersky's Functional Programming course - which was superb. For the lectures on Actors, <a href="http://www.amazon.com/Learning-Concurrent-Programming-Aleksandar-Prokopec/dp/1783281413" target="_blank">Learning Concurrent Programming in Scala</a> has a chapter on Actors which I would recommend to be read first before viewing the lectures. The same is true for Futures.</div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;">- Assignments are completely test driven. That is good for grading, but passing the test is just the first step. Ensuring that your code is written using the finer points of the principles taught is up to you. You might get 10/10 using the automated test grader but your code might not be "correct". I had this experience in the final assignment. This has been pointed out by many in the forums too.</div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;"><br /></div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;">Overall, it's a must-take course if you plan to learn about Reactive Programming.</div><div style="background-color: white; font-family: arial, sans-serif; font-size: 12.8000001907349px;"><br /></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-9306636091122357132014-12-02T12:28:00.003+05:302014-12-02T12:29:42.762+05:30Effective email communication<span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Communication and its various nuances always fascinate me. There are times when I realize, not always too late, that I have failed in communicating what I wanted to convey. It always ends up being a learning experience for me.</span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"><br /></span><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">For most people, the word "communication" seems to remain confined to what one says or writes. But it's far, far more than that.</span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"><br /></span><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">I wanted to share a few tips I have learned about effective <i>email</i> communication over the years. I've picked these up from observation as well as from friends and colleagues. I still commit some of these mistakes when I'm in a hurry but I hope I am getting better.</span><br /><ul><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"> Know your recipients. Tailor your email accordingly. Put yourself in their situation</span></li><ul><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Their awareness of what you're talking about. Do they have prior context and how much? </span></li><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Their environment e.g. Sharing a URL in your email that works only on Chrome (and they use Firefox), or sending URLs that don't work outside your office network.</span></li><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Their focus e.g. Are they likely to single out one out of multiple points in the email and downplay the rest? How do you address any concerns that the recipient might have? Thinking about these beforehand might you save an email iteration or more.</span></li></ul><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Make your intentions clear. If there are actionables, point them out. If you know the owner of the action, point him/her out. If you don't, ask. If it's not an actionable email, mention it (FYI, JFYI) and explain why you are sending the email. </span></li><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Use a meaningful subject line</span></li><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Use To, Cc and Bcc carefully</span></li><ul><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">If you're addressing one or more people in the email body, you can put them in the To field</span></li><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Be careful while Bcc'ing. If the Bcc'ed person does not realize she is Bcc'ed, she might respond to all and then everybody will know, which you might not have intended. If you're the Bcc'ed person, it's upto you to check the email headers and be cognizant of this.</span></li><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Be careful while clicking Reply. You might have meant Reply-All. Gmail/Google Apps Mail have a setting where you can set Reply All as the default.</span></li></ul><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">If the email thread has been going on for sometime, it's helpful to summarize everything, including repeating what has been already said, when a conclusion has been reached. </span></li><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Don't clear the previous content when you respond. People often have to look at the whole thread to regain context.</span></li><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">If the thread has forked off to another topic, or you want to do the forking, change the subject to something appropriate that suits the new topic. </span></li></ul><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"><br /></span><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Somebody said "Communication is about the receiver". If my recipient does not get what I'm trying to convey, <i>I</i> have failed, and not the recipient. This might sound extreme but it's an effective ideal to work towards.</span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-42943700094494279832013-11-23T23:45:00.002+05:302013-11-23T23:46:27.849+05:30Graphite Tip: Disabling data averaging while viewing graphs<span style="font-family: Arial,Helvetica,sans-serif;"><a href="http://graphite.readthedocs.org/en/latest/" target="_blank">Graphite</a>, the superb graphing tool, has gained a lot of popularity lately and with good reason. It's flexible, fairly easy to setup, very easy to use and has a thriving community with plugins for many monitoring systems. It can store any kind of numeric data over time.</span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span><span style="font-family: Arial,Helvetica,sans-serif;">By default, Graphite stores data in WhisperDB, a fixed size database with configurable retention periods for various resolutions. What this means is that you can store higher resolution data (say data for every 5 seconds) for a shorter period of time (e.g. 1 month) and then store the same data at the lower resolution (say for every hour) beyond that time period. The data will be consolidated based on the the method you configure (sum, average). This behaviour of Graphite is well known.</span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span><span style="font-family: Arial,Helvetica,sans-serif;">What is not so well known is that Graphite also does consolidation when you view the graphs. This happens when the number of data points is more than the number of pixels. In such cases, the Graphite graph renderer will consolidate the data into one point using an aggregation function. The default aggregation function is <i>average</i>. So you might end up <a href="https://answers.launchpad.net/graphite/+question/199322" target="_blank">seeing smaller values</a> than you expect.</span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span><span style="font-family: Arial,Helvetica,sans-serif;">Here's an example of a graph where there are more data points than pixels. The actual peak value was a little over 200, but you cannot see it here due to averaging.</span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span><br /><div class="separator" style="clear: both; text-align: center;"><span style="font-family: Arial,Helvetica,sans-serif;"><a href="http://2.bp.blogspot.com/-yOpFBEMHrGY/UpDusD3UREI/AAAAAAAAABk/T8etaWphz9s/s1600/normal-average.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-yOpFBEMHrGY/UpDusD3UREI/AAAAAAAAABk/T8etaWphz9s/s1600/normal-average.png" /></a></span></div><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span><br /><span style="font-family: Arial,Helvetica,sans-serif;">Here is the same graph (same data for the time span) where the image width has been increased<b>*</b> (== more pixels). You can see the peak is almost 200. </span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-GU3hLLTMakY/UpDuzemL28I/AAAAAAAAABs/d55T96EuGr4/s1600/widened-average.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="66" src="http://3.bp.blogspot.com/-GU3hLLTMakY/UpDuzemL28I/AAAAAAAAABs/d55T96EuGr4/s640/widened-average.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Click to view larger</td></tr></tbody></table><br /><span style="font-family: Arial,Helvetica,sans-serif;">Sometimes this behaviour may not be what you want. To see the "actual" data points irrespective of what size your image is, Graphite's URL API provides a property called minXStep. To use it simply add the property as a request parameter (with value 0) in the graph URL. From the <a href="https://graphite.readthedocs.org/en/latest/render_api.html#minxstep" target="_blank">documentation</a>:</span><br /><span style="font-family: Arial,Helvetica,sans-serif;"> <span style="font-size: x-small;"><i> </i></span></span><br /><blockquote class="tr_bq"><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: x-small;"><span style="font-size: small;"><i>To disable render-time point consolidation entirely, set this to <span class="docutils literal"><span class="pre">0</span></span> though note that series with more points than there are pixels in the graph area (e.g. a few month’s worth of per-minute data) will look very ‘smooshed’ as there will be a good deal of line overlap.</i> </span></span></span></blockquote><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span><span style="font-family: Arial,Helvetica,sans-serif;">The same graph with minXStep=0 now looks like this:</span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-deLBQCTkwok/UpDvU9NkvnI/AAAAAAAAAB0/49kP76PFerc/s1600/normal-xstep.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-deLBQCTkwok/UpDvU9NkvnI/AAAAAAAAAB0/49kP76PFerc/s1600/normal-xstep.png" /></a></div><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span><span style="font-family: Arial,Helvetica,sans-serif;">A bit "smooshed" but with the exact data that was collected.</span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><br /><i><span style="font-size: x-small;"><b>*</b> Pass width=x as a request parameter to the graph URL, x in pixels</span></i></span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-45497939109586825372013-09-30T23:51:00.000+05:302013-09-30T23:51:13.657+05:30Revoking private key access to EC2 instances, and other random tips<span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Consider the following scenario</span></span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span><ul><li><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">You have many EC2 instances running production code</span></span></li><li><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Access to those instances is using a passphrase-protected key</span></span></li><li><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">A member of your operations team who has access to the key leaves so you have to change the key. Or, you need to change the existing key as a matter of some internal security policy.</span></span></li></ul><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">How do you do it?</span></span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">You</span></span><br /><ul><li><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Generate a new keypair</span></span></li><li><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Add the public key to the EC2 instances' <login user's home dir>/.ssh/authorized_keys</span></span></li><li><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Remove the old public key from the same authorized_keys file</span></span></li><li><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Done. The old key is useless now.</span></span></li><li><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">This is not actually revocation</span></span></li></ul><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Some things to note about AWS keypairs </span></span><br /><ul><li><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">EC2 metadata for the instance(s) will continue to show the original keypair name it was created with, whatever keys you add or remove from authorized_keys. The original public key may not even exist on the instance anymore, if you have gone through the steps above, but the metadata will still show it. This is because AWS has no way of knowing that you changed the authorized_keys file. </span></span></li><li><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">You can upload keys generated by yourself to the AWS console and they will be available for use while launching EC2 instances. Your generated keys have to be RSA keys of 1024, 2048 or 4096 bits.</span></span></li><li><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">AWS keypairs are said to be confined to a single region. This is true only if you consider the default state of affairs. You can get around it. </span></span></li><ul><li><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">For keys that you generate, you can import them to all the regions you want using the AWS console or the CLI tools. </span></span></li><li><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">For keys that AWS generates, you can take the public key from an EC2 instance launched with that key, and import that in a similar manner to all the regions you want. The private key is available for download when you generate the key.</span></span></li></ul></ul><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-27840284520456030132013-09-27T22:48:00.000+05:302013-09-27T22:52:10.650+05:30Private Cloud Options with Amazon Web Services - Part 1<span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Amazon Web Services is the largest IaaS provider, according to this <a href="http://blogs.gartner.com/lydia_leong/2013/08/21/the-2013-cloud-iaas-magic-quadrant/" target="_blank">Gartner report</a>, in terms of compute capacity. AWS also has a wider geographical presence than other similar companies. </span></span> <span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"> </span></span><br /><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">AWS offers an option to have a private cloud inside their public cloud. You can run this as a small personal cloud, or use one of Amazon's connectivity offerings to connect it securely to your existing infrastructure. </span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">This is an overview of the private cloud options with AWS, followed by an overview of the various connectivity options.<br /><br /><b>Private Cloud Options</b><br />When you launch a regular EC2 instance, it has a public IP address. It is always reachable from the public internet whether you want it or not. You can configure the instance's AWS security group (the inbuilt firewall) to allow access to specific ports only, but this may not serve your security needs. You might want traffic to flow only between your instances and not from the internet.</span></span> <span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br />The obvious way to do this is to not have IP addresses which are reachable from the internet, i.e., use private IP addresses. Which is exactly what <a href="http://aws.amazon.com/vpc/" target="_blank">VPC</a> offers.</span></span><br /><div><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><b>IP Ranges</b><br />A <a href="http://aws.amazon.com/vpc/" target="_blank">VPC</a> is like a private network inside Amazon's cloud where you can create smaller subnets and instances inside them. While creating a VPC, you'll need to define the range of IP addresses that the VPC will cover. </span></span> </div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span></div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><b>Subnets</b> </span></span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">The basic unit of a VPC is a subnet - a logical network where you can create instances, define the range of private IP addresses that the instances inside it will have and create routing tables to define how traffic is routed to and from the subnet.</span></span></div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-iuRcY2iqTzc/UkW7prZam0I/AAAAAAAAABE/K5xDV6WeIgg/s1600/Screenshot+from+2013-09-27+22:35:07.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="182" src="http://1.bp.blogspot.com/-iuRcY2iqTzc/UkW7prZam0I/AAAAAAAAABE/K5xDV6WeIgg/s320/Screenshot+from+2013-09-27+22:35:07.png" width="320" /></a></div><div></div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><b>Kinds of Subnets</b></span></span></div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">There are two kinds of subnets you can create inside a VPC</span></span></div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Private : EC2 instances created inside it cannot talk to the internet and vice versa.</span></span> </div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Public : EC2 instances created inside it can access the internet and can also be made accessible from the internet.<br /><br />Private and public are just names and not inbuilt properties. What actually makes them "private" and "public" are the routing tables you create and assign to the subnets. So you must first create the subnets, then create the tables, assign them to the subnets and finally give them descriptive names. If you use the VPC wizard, it will do this for you. You can create multiple subnets of each type.</span></span> </div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span></div><div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><b>Communication between a public subnet and the internet</b></span></span></div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">If you want your instances to access the internet, you have an option of adding an "internet gateway" to a subnet. The internet gateway here is an AWS abstraction. You would add this to your public subnet (or subnets). Once you assign a gateway, you must assign an elastic IP to an instance inside that subnet. This instance is the one that would be able to communicate with the outside world.</span></span></div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span> <span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"></span></span></div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span></div><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-UwCxLmNfvaA/UkW8FXW8JtI/AAAAAAAAABM/rHWRLr5063E/s1600/Screenshot+from+2013-09-27+22:35:15.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="151" src="http://3.bp.blogspot.com/-UwCxLmNfvaA/UkW8FXW8JtI/AAAAAAAAABM/rHWRLr5063E/s320/Screenshot+from+2013-09-27+22:35:15.png" width="320" /></a></div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span></div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">VPC places a limit on the number of elastic IPs (5). If you have many instances which need to access the internet, you would put all of them behind a single instance with an EIP instead of assigning each an EIP, and use NAT to access the internet from the "hidden" instances.</span></span> </div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /><b>Communication between a private and a public subnet</b><u><br /></u></span></span></div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Setting up communication between a private and a public subnet is a straightforward configuration in the routing table.<br /><u><br /></u>A typical example of using both private and public subnets in a VPC is from the AWS documentation:</span></span> </div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span></div><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-LxFz6GrB8ZI/UkW8MRs5JGI/AAAAAAAAABU/wdAqhNwRt8A/s1600/Screenshot+from+2013-09-27+22:35:20.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="149" src="http://2.bp.blogspot.com/-LxFz6GrB8ZI/UkW8MRs5JGI/AAAAAAAAABU/wdAqhNwRt8A/s320/Screenshot+from+2013-09-27+22:35:20.png" width="320" /></a></div><div></div><div><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Here, the database servers are extra-secure inside a private subnet, while the webservers are in the public subnet, as they have to serve traffic to end users. </span></span></div><div></div><div><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">The "private" nature of a VPC is not limited to the network alone. Inside a VPC, you have the option of launching a regular EC2 instance, which is a virtual machine on a host shared with other guest VMs. You can also choose to launch a dedicated instance - which is a truly dedicated machine used only by your instance, giving you isolation at the hardware level as well. Costs are <a href="http://aws.amazon.com/dedicated-instances/" target="_blank">slightly higher</a> for dedicated instances.</span></span></div><div></div><div><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">A VPC lets you setup your own private cloud with isolation at the hardware and the network levels. I'll explore the various connectivity options between VPCs and your own datacenter in the next post.</span></span></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-26779354688061590692013-05-17T20:30:00.000+05:302013-05-17T20:33:59.625+05:30Book Review : The Art of Scalability<span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><b>About the author</b>: Theo Schlossnagle is the founder and CEO of <a href="http://www.omniti.com/" target="_blank">OmtiTI</a>. </span></span><br /><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><b>The Book</b>: </span></span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">This book aims to be a comprehensive, technology stack-agnostic compendium of strategies and guidelines to achieving scalability objectives for internet applications. It is quite thin (262 pages) and came out in 2007, when the DevOps meme was not around in its current form.</span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><b> </b></span></span></span></span><br /><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><b>Why I like this book</b>: </span></span> </span></span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">I like this book because it's oriented towards building a solid foundation on topics related to scaling. Compare this book with '<a href="http://www.amazon.com/Web-Operations-Keeping-Data-Time/dp/1449377440" target="_blank">Web Operations: Keeping the Data on Time</a>' (published in late 2010), and you'll find the <span style="font-size: small;">book under discussion</span> to be more grounded in fundamental principles, and the latter more oriented towards new trends. Now there's nothing wrong with the 'latest-trend' books, but it's better if one reads this kind first to get a good grounding.</span></span><br /><br /><span style="font-size: small;"><b>Overview of Chapters:</b></span> <br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">The first three chapters cover basic principles, managing release cycles and operations teams. </span></span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">A big part of chapter 4 is devoted to explaining the difference between high availability and load balancing. There's no coverage of Cloud based options here – this is for you if you manage your own datacenters. Also, cloud based options will invariably be tied to specific vendors. Different HA options are considered with almost academic rigour. </span></span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Chapter 5 examines load balancing options at different layers of the OSI network stack.</span></span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Chapter 6 is a mini-guide to building your own Content Delivery Network. From calculating your expected traffic, cost estimates, inter-node synchronization in a cluster to choosing the OS and having an HA network configuration – it's an interesting journey. It brings out the challenges which are invisible to most of us who push our static content to a third party CDN and forget about it. There's a section on DNS issues as well covering <a href="http://serverfault.com/questions/14985/what-is-anycast-and-how-is-it-helpful" target="_blank">Anycast</a>.</span></span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Chapter 7 covers five caching techniques. True to the general theme of the book, it does not talk about specific technologies but about theory that can be studied and applied to the problem at hand. An example of speeding up a news website is used to illustrate how to deploy and tune memcached (for that specific site's design).</span></span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">In Chapter 8, we see an overview of distributed databases, including an overview of different database<span style="font-size: small;"> </span>replication strategies. Managing, storing, aggregating and parsing logs is a challenge we all face – this is covered in Chapter 9. This chapter is dated now as there have been many advances on this topic.</span></span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Overall, a must-have for anybody who is interested or works in scaling internet facing applications.</span></span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Amazon US URL: <a href="http://www.amazon.com/Scalable-Internet-Architectures-Theo-Schlossnagle/dp/067232699X">http://www.amazon.com/Scalable-Internet-Architectures-Theo-Schlossnagle/dp/067232699X</a></span></span><br /><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Indian bookstores: <a href="http://isbn.net.in/9788131706114">http://isbn.net.in/9788131706114</a></span></span>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-513893015191696787.post-43743084963947116392013-05-04T17:26:00.002+05:302013-05-04T21:27:22.889+05:30Thoughts on "A Note on Distributed Computing"<span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"><b><i>A Note on Distributed Computing by Jim Waldo, Geoff Wyant, Ann Wollrath, and Sam Kendall</i></b> is a widely cited paper. I have been reading and trying to understand it for sometime. It's available here - <a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.7628" target="_blank"><span style="font-size: small;">http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.7628</span></a></span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"><br />The title of the paper is innocuous but it's much more than a "note". It analyzes the key differences between local and distributed computing, and explains why attempts to unify their programming models are misguided because of the fundamental differences underlying them.</span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"><br />The authors used to be part of the erstwhile Sun Microsystems when they wrote it - it dates from 1994. Later some of them were members of the JINI technology team and also wrote RMI, and if you look at the Java RMI source code, you can see some of their <a href="http://javasourcecode.org/html/open-source/jdk/jdk-6u23/java.rmi/RemoteException.java.html" target="_blank">names</a>.</span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"><br /></span><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">But in 1994 when this paper was written, Java had not emerged yet. There was no J2EE and CORBA was still young.</span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"><br />I wish to share my thoughts after reading it, and the realization that the opinions expressed in it influenced the design of Java's RMI.</span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"><br /><b>Unification</b></span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Briefly, unification = unification of the local and the distributed programming models. Note that we are talking about distributed object oriented systems here. </span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"><br />The unification attempts assume that objects are essentially of a single top level type (like in Java), which might span different address spaces on the same or different machines (like different JVMs on the same or different machines in the case of Java) and they can communicate in the same way irrespective of where they are located. In other words, location (same JVM versus another JVM in another country) is merely an implementation detail that can be abstracted away behind the interfaces used to communicate between two objects without any side effects. </span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"><br />Such a (hypothetical) system would have the following characteristics</span><br /><ul></ul><ol><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Program functionality is not affected by the location of the object on which an operation has been invoked. Or viewing it from a slightly higher level, there is a single design as to how a system communicates irrespective of whether it's deployed in one address space or in multiple ones.</span><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"> </span></li><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Maintenance and upgrades can be done to individual objects without affecting the rest of the system.</span><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"> </span></li><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">There is no need to handle failure and performance issues in the system design.</span></li><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Object interfaces are always the same regardless of the context (i.e. remote or local) </span></li></ol><ul></ul><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">The authors contend that all these statements are flawed. I'll not attempt to go into those details - the paper explains them well.</span><br /><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">The paper then goes onto examine the 4 areas where local and distributed computing differ drastically:<br /> Latency</span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"> Memory Access</span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"> Partial Failure</span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"> Concurrency</span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"><br />Those of us who have worked on distributed enterprise and internet software have come across these. These 4 differences cannot be papered over to present a 'unified' view of objects which lie on different machines.</span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"> </span> <br /><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;"><b>Java RMI</b></span><br /><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">If you look at RMI, you can see its design influenced by the assumption that the above 4 points are invalid.</span><br /><ul><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">"Remote" objects have to extend the java.rmi.Remote interface</span><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">. <span style="font-size: small;">"Remote" o<span style="font-size: small;">bjects - object<span style="font-size: small;">s that can be invoked from another JVM - a<span style="font-size: small;">re different from local objects.</span></span></span></span></span></span></li><li><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Remote (inter-JVM) method calls have to explicitly handle the java.rmi.RemoteException, which is a checked exception, thus highlighting the fact that a distributed call is subject to modes of failure that are non-existent in a local call. In fact, it <span style="font-size: small;">extends java.io.IOEx<span style="font-size: small;">ception and the <a href="http://javasourcecode.org/html/open-source/jdk/jdk-6u23/java.rmi/RemoteException.java.html" target="_blank">javadoc</a> is expl<span style="font-size: small;">icit about network <span style="font-size: small;">issues "</span></span></span></span></span><i><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">is the common superclass for a number of communication-related exceptions that may occur during the execution of a remote method call</span></span></i>".</li></ul><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Let's look at #2 again. From the paper:</span><br /><blockquote class="tr_bq"><span style="font-size: small;"><i><span style="font-family: Arial,Helvetica,sans-serif;">"As long as the interfaces between objects remain constant, the implementations of those objects can be altered at will".</span></i></span></blockquote><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Premonition of SOA, anyone? This concept would be familiar today to anybody who is acquainted with the fundamental principles of service oriented system design (replace 'object' with 'service'). But since these statements are challenged and refuted later in the paper, the question naturally arises - how come SOA is successful?</span><span style="font-size: small;"> </span><br /><br /><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">SOA assumes that things are independent and distributed services, and any invocation of a service assumes that there are failure modes which exist because of the <span style="font-size: small;">communication's distributed</span> nature. This builds on the same RMI concept as having to explicitly throw RemoteException when making a remote (distributed) call. This same concept is taken into consideration while writing any SOA system, which is another way of saying that the authors of the paper were correct.</span></span> <br /><span style="font-size: small;"><br /></span><span style="font-family: Arial,Helvetica,sans-serif; font-size: small;">Note: A short and readable summary of Java RMI is to be found in Jim Waldo's book <a href="http://www.amazon.com/Java-Good-Parts-Jim-Waldo/dp/0596803737" target="_blank">Java: The Good Parts</a>.</span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-85832710908415586112013-04-28T17:52:00.003+05:302013-04-28T20:45:59.551+05:30"Upgrading" to Fedora 18<span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">I have been running Fedora 16 on my work laptop. It was EOL'ed early this year, which means no more upgrades, including for things like Firefox. There was no option but to upgrade. I had two choices - opt for something with long term support like Ubuntu LTS or try the new Fedora (and try the newer one in 6 months).</span></span></span><br /><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span></span></span><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">I opted for the latter, since I've been using Fedora for a while, hoping that I would not have to do a Windows-style post installation cleanup. </span></span></span><br /><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span></span></span><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">No such luck. The things that were broken, still are.</span></span></span><br /><span style="background-color: white;"><br /></span><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">Some highlights from the experience:</span></span></span><br /><span style="background-color: white;"><br /></span><b><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">Nepomuk, Akonadi</span></span></span></b><br /><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">Disable them? Sure. They are disabled in System Settings, but insist o<span style="font-size: small;">n</span> starting up anyways. Uninstall them? Not possible. They're so tightly coupled with KDE that uninstalling them uninstalls all of KDE. The developers don't seem to be <a href="https://www.google.co.in/search?client=ubuntu&channel=fs&q=how+to+stop+akonadi" target="_blank">listening</a> to the users here. </span></span></span><br /><span style="background-color: white;"><br /></span><b><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">Disk space</span></span></span></b><br /><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">My installation ran out of disk space 20 minutes after I rebooted post-installation. There seemed to be some continuous process in the background which<span style="font-size: small;"> was</span> eating up space. Some investigation identified the culprit. </span></span></span><br /><blockquote class="tr_bq"><span style="background-color: white;"><span style="font-size: xx-small;"><span style="font-family: "Courier New",Courier,monospace;">[talonx@****** apps]$ pwd<br />/home/talonx/.kde/share/apps<br />[talonx@****** apps]$ find . -type f -size +50000k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'<br />./nepomuk/repository/main/data/virtuosobackend/soprano-virtuoso.log: 151G<br />./nepomuk/repository/main/data/virtuosobackend/soprano-virtuoso.db: 68M</span></span></span></blockquote><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">Yes, it created a log file of 151G within 20 minutes. What kind of application does that? What about basic stuff like log file rotation?</span></span></span><br /><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span></span></span><b><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">PackageKit</span></span></span></b><br /><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">Another of those which <a href="http://forums.fedoraforum.org/showthread.php?t=273225" target="_blank">does not go away</a>, and causes endless irritation. </span></span></span><br /><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;"><br /></span></span></span><span style="background-color: white;"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">Fedora is not something that I would prescribe to new Linux users. Others have pointed out that its instability and some features are probably the <a href="https://ask.fedoraproject.org/question/7683/is-there-a-fedora-stable-lts/" target="_blank">result of staying at the cutting edge</a>. Granting that, it<span style="font-size: small;"> remains </span>difficult to get it to a state where <span style="font-size: small;">even people like</span> software developers can use it to be productive.</span></span></span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-54271147022422751642012-12-08T21:39:00.001+05:302012-12-08T21:39:57.666+05:30Experiences in building a home NAS - Part 1Some months back I had this email conversation with a friend about the best strategy for storing movies/music/photos at home. I was running out of space, again. One of my external 1TB drives had crashed. <br /><br />Hard drives fail all the time. To backup the hard drive, you need another. Multiply that by the number of drives you have. The bigger the hard drive which crashed, the bigger the psychic shock. And so on. The buy-another-1TB-drive-strategy was not working.<br /><br />My friend had done some research on this topic, and he suggested building a NAS box out of commodity hardware. We exchanged mails back and forth and researched a lot for a month. There's a huge internet community doing the same thing so finding information was not a problem. An option which seemed attractive was combining a small server from HP with my own storage and RAM. This was the <a href="http://h10010.www1.hp.com/wwpc/uk/en/sm/WF06a/15351-15351-4237916-4237917-4237917-4248009.html?dnr=1" target="_blank">HP ProLiant MicroServer N36L/N40L</a>.<br /><br /><b>The hardware</b><br /><br />I finally went with this to avoid having to choose the non-storage hardware myself, being no expert in it. The specs for the N36L model were decent enough - <br /><ul><li>AMD Athlon II Neo (dual core) 1.8 GHz</li><li>1 GB included RAM (max 8 GB)</li><li>Seagate 160 GB</li><li>Gigabit ethernet </li></ul>The RAM and HDD were of course, not enough. But the rest of it was. It came with 4 drive bays - to put my own hard disks in.<br /><br />There were other factors in choosing this<br /><ul><li>I did not want a hardware RAID controller - it ties you down to the RAID controller.</li><li>I did not care for more storage bays.</li><li>It has 7 USB ports, with one internal which can host a flash drive to boot from, thus saving the other drive bays for storage!</li><li>To top it all, it was good looking.</li></ul><br /><b>The software</b><br /><br />That's the hardware. What about the OS? NAS software? Like I said, my friend had already done some research, and I got the pointers from him. <a href="http://www.freenas.org/" target="_blank">FreeNAS</a> is a superb option if you're building your home NAS. It's based on FreeBSD and supports the ZFS file system. ZFS was conceived and implemented at the erstwhile Sun Microsystems. The virtues of this filesystem - I'll just point you to the <a href="http://en.wikipedia.org/wiki/ZFS#Features" target="_blank">documentation</a>. Resilvering (auto repair) and copy-on-write are two of them.<br /><br /><b>Software RAID</b><br /><br />ZFS supports various software RAID options, including <a href="http://en.wikipedia.org/wiki/Non-standard_RAID_levels#RAID-Z" target="_blank">RAID-Z</a>. RAID-Z1 (the first level) essentially gives you the ability to survive one hard disk crash if you have 3.<br /><br />With RAID-Z1, if you have 3 x 1 TB drives, you will use 2 x 1 TB for data and 1 x 1 TB for parity information. Even if one of the hard disks crashes, you can add a new one and you're fine. You use 1 TB to pay for recoverability. There are other configurations if you want more redundancy, like being able to recover if you lose more than one drive at the same time - which are more expensive as they need more disks.<br /><br /><b>Buying the hardware</b><br /><br />I bought the ProLiant from a local dealer after scouring the Hyderabad classified pages. The HDD (3 x 1TB) and 4 GB of RAM I bought online, from Flipkart. Here are two bits of learning if you're doing the same<br /><ul><li>Buy ECC RAM if you want surefire data integrity.</li><li>Buy more hard disk space than you need now - you won't regret it. It will cost a bit more, but it's better than the alternative. The alternative would be to recreate all your ZFS volumes on the new (bigger disks). With my configuration, I'll have to buy 3 x (whatever GB) to upgrade my setup. Of course, the third option is that you don't go this HP microserver way at all, and choose a bigger box (custom built or otherwise) which supports more drives.</li></ul>It's a NAS, so I needed a network switch, which I bought from eBay. <a href="http://www.tp-link.com/en/products/details/?model=TL-SG1008D" target="_blank">This</a> had good reviews, had 8 ports and supported Gigabit ethernet. Plus some CAT6 cables.<br /><br /><b>Putting it all together</b><br /><br />The HP came with 4 drive bays, with one small (160 GB) HDD, where I installed FreeNAS. The other three went for storage (and parity). The RAM worked without a hitch, but that was because I had ensured it was compatible (see these <a href="http://n40l.wikia.com/wiki/HP_MicroServer_N40L_Wiki" target="_blank">hardware compatibility</a> <a href="http://www.overclockers.com.au/wiki/HP_Microserver" target="_blank">lists</a>).<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-3NG-KfoxQcE/UMNdjwy12WI/AAAAAAAAAAQ/37EBe8jLYBk/s1600/installation.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="186" src="http://1.bp.blogspot.com/-3NG-KfoxQcE/UMNdjwy12WI/AAAAAAAAAAQ/37EBe8jLYBk/s320/installation.jpeg" width="320" /></a></div><br />(All those cables were to connect my desktop monitor, keyboard and DVD drive to the HP server).<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-ZdnYF2iIdIg/UMNd099n_mI/AAAAAAAAAAY/7KDjhG5E8To/s1600/innards.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="234" src="http://1.bp.blogspot.com/-ZdnYF2iIdIg/UMNd099n_mI/AAAAAAAAAAY/7KDjhG5E8To/s320/innards.jpeg" width="320" /></a></div><br /><br /><br />Internal view - the 4 drive bays in vertically placed in front.<br /><br /><br />The FreeNAS interface is easy to use, so setting up ZFS volumes was a breeze.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-jZMHYgcHfwM/UMNfkq0XRnI/AAAAAAAAAAg/yMc_QiqjL3Q/s1600/freenas.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="204" src="http://3.bp.blogspot.com/-jZMHYgcHfwM/UMNfkq0XRnI/AAAAAAAAAAg/yMc_QiqjL3Q/s320/freenas.jpeg" width="320" /></a></div><br /><br />It also comes with a Cacti-like interface which lets you view OS metrics. <br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-5PNwl1zK5-8/UMNfrrGByrI/AAAAAAAAAAo/wXcJoJnQgJg/s1600/reporting.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="http://4.bp.blogspot.com/-5PNwl1zK5-8/UMNfrrGByrI/AAAAAAAAAAo/wXcJoJnQgJg/s320/reporting.jpeg" width="278" /></a></div><br /><br /><br />To complete the setup, I connected the 8 port switch to my ISP's router, and plugged in my desktop and the HP to the switch. FreeNAS lets you create NFS shares for the data stored on the NAS which I can then access as a network mounted volume.<br /><br />There really are a lot of resources about setting up a home NAS in this configuration. Some of the ones that I found helpful are <br /><ul><li><a href="http://www.avforums.com/forums/networking-nas/1431514-hp-proliant-microserver-n36l-owners-thread.html" target="_blank">HP ProLiant MicroServer N36L owners' thread</a></li><li><a href="http://forums.overclockers.com.au/showthread.php?t=958208" target="_blank">Another owners' thread</a> </li><li><a href="http://www.tenniswood.co.uk/technology/windows/review-hp-microserver-for-windows-home-server/" target="_blank">A review of the machine</a></li><li><a href="http://n40l.wikia.com/wiki/HP_MicroServer_N40L_Wiki" target="_blank">N40L wiki</a> (it's not very different from the N36L that I have)</li><li><a href="http://www.overclockers.com.au/wiki/HP_Microserver" target="_blank">Another hardware compatibility list</a></li><li>RAID-Z? <a href="https://blogs.oracle.com/roch/entry/when_to_and_not_to" target="_blank">Here you go</a></li><li><a href="https://blogs.oracle.com/timc/entry/demonstrating_zfs_self_healing" target="_blank">ZFS self-healing</a></li></ul><br />This does not complete the NAS setup. Performance testing has to be done. And also, backups. I'll write about these in the next post.<br /><ul> </ul><br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-27666919868064764662012-11-19T23:40:00.000+05:302012-11-19T23:40:00.454+05:30Nagle's algorithm and delayed acks......don't work well together. I finally understood why from Richard Steven's <a href="http://www.unpbook.com/" target="_blank">UNP</a> book.<br /><br />In a nutshell,<br /><br /><blockquote class="tr_bq"><span style="font-family: "Courier New",Courier,monospace;">S(ender) sends a packet, and cannot send the second one if the second one's size is less than the MSS (maximum segment size) until the first is acknowledged (Nagle). R(eceiver) receives the first packet, but cannot acknowledge it until the receiver app tries to send data on which the ACK can piggyback (Delayed ACK). S waits, R waits - until the delayed ACK timer on R times out.</span><br /><span style="font-family: "Courier New",Courier,monospace;"></span><br /><span style="font-family: "Courier New",Courier,monospace;"></span></blockquote><br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-90504192044862138542012-11-15T22:46:00.001+05:302012-11-15T22:48:25.930+05:30No more PermGen in the JDK<span style="font-family: inherit;"><span style="color: black;">PermGen space is familiar to anybody who has debugged memory issues in large JEE applications. Starting with JDK 8, it is being removed. The <a href="http://openjdk.java.net/jeps/122" target="_blank">JDK Enhancement Proposal</a> says "remove", but it's more like it's being moved to native memory outside the JVM.</span></span><br /><span style="font-family: inherit;"><span style="color: black;"><br /></span></span><span style="font-family: inherit;"><span style="color: black;">The goal as mentioned in the JEP is "to remove the need to tune the size of the permanent generation."</span></span><br /><span style="font-family: inherit;"><span style="color: black;"><br /></span></span><span style="font-family: inherit;"><span style="color: black;">The beginnings of the change are already in JDK7, with interned strings being no longer part of the PermGen, but of the main Java heap - <a href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6962931" target="_blank">http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6962931</a></span></span><br /><span style="font-family: inherit;"><span style="color: black;"><br /></span></span><span style="font-family: inherit;"><span style="color: black;">There's a summary of changes in the OpenJDK mailing list</span></span><br /><span style="font-family: inherit;"><span style="color: black;"><a href="http://mail.openjdk.java.net/pipermail/hotspot-dev/2012-September/006679.html">http://mail.openjdk.java.net/pipermail/hotspot-dev/2012-September/006679.html</a></span></span><br /><span style="font-family: inherit;"><span style="color: black;"><br /></span></span><span style="font-family: inherit;"><span style="color: black;">What does this mean for Java developers? Well, at least the following</span></span><br /><span style="font-family: inherit;"><span style="color: black;">- There will no longer be any need to set the sizes of the Perm Gen space with JVM startup parameters.</span></span><br /><span style="font-family: inherit;"><span style="color: black;">- The class metadata previously stored in the PermGen will now be stored in a native memory space called "Metaspace".</span></span><br /><span style="font-family: inherit;"><span style="color: black;">- The default maximum size of the metaspace will be limited only by the available memory on the machine. However, it can be limited by using the MaxMetaspaceSize option.</span></span><br /><span style="font-family: inherit;"><span style="color: black;">- It's not entirely clear yet as to how debugging tools like jmap and jhat will change, if at all, as a result of this, or if it will bring in more complications.</span></span><br /><span style="font-family: inherit;"><span style="color: black;"><br /></span></span><span style="font-family: inherit;"><span style="color: black;">And of course, if you have classloader-leaks-based OutOfMemory problems in your application, they are not magically going away. They will just be transferred to another space with more memory than is allocated to your JVM now.</span></span><br /><span style="font-family: inherit;"><span style="color: black;"><br /></span></span><span style="font-family: inherit;"><span style="color: black;">I could not find any reference to the origin of the term "metaspaces" except in this paper (PDF) - <a href="http://www.ssw.uni-linz.ac.at/General/Staff/TS/optimized_class_metadata_memory_management_in_a_jvm.draft.pdf">http://www.ssw.uni-linz.ac.at/General/Staff/TS/optimized_class_metadata_memory_management_in_a_jvm.draft.pdf</a></span></span><br /><span style="font-family: inherit;"><span style="color: black;">Familiarity with the JVM's memory management strategies is required to understand it - which I don't have currently - so it's on my to-read list as of now.</span></span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-51923406338593822932012-10-20T19:39:00.001+05:302012-11-15T22:48:40.480+05:30Lessons learned while managing technical operations<span style="font-size: small;"><span style="font-family: inherit;"><span style="color: black;"><span id="internal-source-marker_0.6879697904141526" style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"> for a cloud based SaaS product, and which might be useful to you if you’re doing the same.</span></span></span></span><br /><span style="font-size: small;"><span style="font-family: inherit;"><br /></span></span><ul style="margin-bottom: 0pt; margin-top: 0pt;"><li style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="font-size: small;"><span style="font-family: inherit;"><span style="color: black;"><span style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">There is no substitute to knowing your fundamentals. Whatever you’re managing - your own datacenter or a suite of apps on a public cloud - you have to know your Operating Systems, your Computer Networking, your Linux, your VMs.</span></span></span></span></li><li style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="font-size: small;"><span style="font-family: inherit;"><span style="color: black;"><span style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Know your tools. Find out what tools you need to monitor, maintain and debug your systems. Know how they work, keep up with updates and play with them often. It will save you time when the crisis hits.</span></span></span></span></li><li style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="font-size: small;"><span style="font-family: inherit;"><span style="color: black;"><span style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Know one editor and know it well - vim, emacs or other. Know all common shortcuts, complicated copying pasting routines, tips and tricks - in times of crisis, every second counts.</span></span></span></span></li><li style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="font-size: small;"><span style="font-family: inherit;"><span style="color: black;"><span style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Learn a little every day. Share what you learn even if you think nobody’s listening. Soon you’ll find like minded people you can share ideas with.</span></span></span></span></li><li style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="font-size: small;"><span style="font-family: inherit;"><span style="color: black;"><span style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Visibility to other teams of what you’re doing is very important. Graph it, present it, blog and talk about it.</span></span></span></span></li><li style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="font-size: small;"><span style="font-family: inherit;"><span style="color: black;"><span style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Try to fill your team with the right people. The best people in technical operations have an eye for detail who do not lose sight of the big picture. They are good split-second decision makers and are experts in prioritizing in times of crises. And of course, they know their stuff or are smart enough to figure it out if they don’t.</span></span></span></span></li><li style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="font-size: small;"><span style="font-family: inherit;"><span style="color: black;"><span style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Know your industry. Study what others are doing, and why.</span></span></span></span></li><li style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="font-size: small;"><span style="font-family: inherit;"><span style="color: black;"><span style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Keep up to date. Know what is new in your field - subscribe to the best newsletters, RSS feeds, podcasts and conferences. There is a lot of noise, so take out the time to sift to the useful parts, adopt what is good for your operations and forget the rest. </span></span></span></span></li><li style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="font-size: small;"><span style="font-family: inherit;"><span style="color: black;"><span style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Keep an open mind. Fads will come and go, old ideas will be repackaged and sold with a new coating every few years. Whatever be the case, keep up with trends - they always have something to teach.</span></span></span></span></li><li style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="font-size: small;"><span style="font-family: inherit;"><span style="color: black;"><span style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;">Know your organization’s business. Interface and build relationships with all teams. If you cut away the trappings of the DevOps movement, the most important point that remains is collaboration. How you achieve it depends on you.</span></span></span></span></li></ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-1464854033575405612012-10-12T21:21:00.005+05:302012-11-15T22:49:07.516+05:30DevOps Resources<span style="color: black;"><span style="font-family: inherit;">I have been following the DevOps "movement" since its inception. Like any cultural meme that has value it has led to thousands of blogs, podcasts and now books on topics that are related directly and indirectly to it. </span></span><br /><span style="color: black;"><span style="font-family: inherit;"><br /></span></span><span style="color: black;"><span style="font-family: inherit;">I've added a page on my blog linking to some podcasts that I have been following (on and off for some them), which might be of interest to somebody working in Technical Operations, Infrastructure, System administration or managing and architecting a cloud hosted product. It's linked from the top of my blog header - <a href="http://code.deepinspace.net/p/blog-page.html" target="_blank">and also from here</a>.</span></span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-69826945250146054672012-10-10T22:19:00.000+05:302013-09-19T14:41:28.305+05:30Multiuser SFTP server setup - the solution<span style="color: black;">I had to setup an SFTP server on an EC2 instance recently, with multiple users chroot-ed into their own directories (with access to only those directories), and a <i>different</i> set of ssh-enabled users, with key based authentication for sftp as well as ssh.</span><br /><span style="color: black;"><br /></span><span style="color: black;">My first instinct was to do a Google search. Many links came up, none of which solved the complete problem. Some of them did not work (different Linux distro/version) and some ended up disabling ssh when I got sftp working.</span><br /><span style="color: black;"><br /></span><span style="color: black;">I finally found this blog post -</span><br /><span style="color: black;"><br /></span><span style="color: black;"><a href="http://blog.famzah.net/2011/02/03/secure-chroot-remote-file-access-via-sftp-and-ssh/">http://blog.famzah.net/2011/02/03/secure-chroot-remote-file-access-via-sftp-and-ssh/</a></span><br /><span style="color: black;"><br /></span><span style="color: black;">It's the only set of instructions that actually worked, with all the constraints mentioned above.</span><br /><span style="color: black;">For the record, the OS was Ubuntu 12.04 LTS. An additional step you need to take on this OS is to disable apparmor, or the ssh stops working after a reboot. I am not a Linux wizard, so I don't know yet why this happens.</span><br /><span style="color: black;"><br /></span><span style="color: black;">On a related note, it turns out that a common mistake that many make is confusing FTP over SSL/TLS with SFTP. FTP over SSL is just FTP over a secure connection, while SFTP is a completely different protocol, with the file transfer happening over an ssh connection.</span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-90860288093102752522012-05-22T23:46:00.000+05:302012-11-15T22:49:20.203+05:30Visualizing EC2<div dir="ltr" style="text-align: left;" trbidi="on"><span style="color: black;"><span style="font-family: inherit;">I hacked up a small application over the weekend to visualize EC2 instances at one place. Python/boto/web.py does the work of retrieving the data and JS (GraphDracula) displays it.</span></span><br /><span style="color: black;"><span style="font-family: inherit;"><br /></span></span><span style="color: black;"><span style="font-family: inherit;">The need for this rose at work - where we use EC2 to host our infrastructure. The original intention behind it was an easier way to view things like</span></span><br /><ul><li><span style="color: black;"><span style="font-family: inherit;">Which availability zone has which instances</span></span></li><li><span style="color: black;"><span style="font-family: inherit;">Open ports between security groups</span></span></li><li><span style="color: black;"><span style="font-family: inherit;">Which security group has which instances</span></span></li><li><span style="color: black;"><span style="font-family: inherit;">and so on</span></span></li></ul><span style="color: black;"><span style="font-family: inherit;">Not all of this has been done yet, but will be.</span></span><br /><span style="color: black;"><span style="font-family: inherit;">Here it is on Github - <a href="https://github.com/talonx/ec2viz" target="_blank">https://github.com/talonx/ec2viz</a></span></span></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-77653528327839884362011-08-08T00:42:00.000+05:302012-06-17T14:10:02.285+05:30The operational mentality in software development<div dir="ltr" style="text-align: left;" trbidi="on"><a href="http://velocityconf.com/velocity2011/public/schedule/detail/20406" target="_blank">A talk by Theo Schlossnagle</a> spurred this line of thought. The description of the talk is "about the evolution of a career in web operations", but he talks more about the importance of thinking operationally by developers. In other words, he takes a different stance about the meaning of DevOps than what is prevalent. DevOps is usually described as increased collaboration between development and operations, with knowledge sharing between the two groups leading to better delivery. <br /><br />Theo Schlossnagle is of the view that developers need to take the operational view when writing code. It's a very valid point of view and something that I suspect most of us overlook. <br /><br />I'm going to expand on what he said and share my ideas on that. <br /><br />Let's take the software you're writing now (assuming it's web software). Is it operable?<br /> It probably does atleast these things<br /> <br /><ul><li>Fulfills your requirements document<br /> </li><li>Passes your unit tests and integration tests<br /> </li><li>The UI is usable and responsive</li></ul><br />But is it operable? Once it's deployed, can it survive unprecedented load? Fringe cases? Subsystems going down? Third party services it depends on becoming unavailable?<br /> <br /> And present a front of graceful degradation as it does all this?<br /><br /><br /><b>Selective Failure</b><br />Most of the time, we stress systems before deployment, using load tests to simulate real world conditions. That takes care of one aspect. But most of us don't think of failures of selective systems, especially when the system is distributed and its components interact in complex ways. The latter is true of most big web applications.Handling selective subsystem failures is not purely an operations responsibility. The application has to be written keeping selective failure in mind. <br /><br />In the video, Theo brings up an analogy with security. Security is not a feature, but a way of thinking.<br />Sanitizing user input before putting it in a database query is not a feature.<br />Not allowing access to your internal web services is not a feature.<br />These are security restrictions you automatically think of when you develop.<br /><br /><br />In the same way, operational thinking should not be a something postponed till deployment while writing code. It should be de rigueur in the design and development process.</div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-91385338826707899492011-07-21T03:52:00.000+05:302012-06-17T14:10:54.128+05:30Some string matching<div dir="ltr" style="text-align: left;" trbidi="on"><a href="http://www.facebook.com/careers/puzzles.php?puzzle_id=17" target="_blank">This puzzle</a> from Facebook's engineering puzzles page is a good introduction to string matching.<br />A solution is possible using an algorithm called <a href="http://en.wikipedia.org/wiki/Levenshtein_distance" target="_blank">Levenshtein distance</a>.<br /><br />Let's take the example on the puzzle page -<br /><blockquote><i>TIHS SENTENTCNES ISS NOUT VARRRY GOUD</i></blockquote><br />It's a munged up version of<blockquote><i>THIS SENTENCE IS NOT VERY GOOD</i></blockquote>The original problem statement involves minimizing the score (number of changes necessary) to transform each word in the munged up sentence into words that are in a predefined <a href="http://www.facebook.com/jobs_puzzles/twl06.txt" target="_blank">wordlist</a>. It says nothing about grammatical correctness of the result.<br /><br />So going by that, the score and the output sentence from my <a href="https://github.com/talonx/python-scribbles/blob/master/fb/breathalyzer.py" target="_blank">implementation</a> are respectively, 8 and<br /><blockquote><i>TICS SENTENCES IDS NOT VAGARY GAUD</i></blockquote><br />From the standpoint of the original problem, this solution is correct. <br />But not from a grammatical point of view. After some poking around, I realized it's not possible for this program to spew out the correct version of the sentence without it being aware of English grammar rules. <br /><br /><br /><b>P.S. </b>Without getting into grammar correction, of which I have no idea how to implement, I made a small change in the program as an experiment. The <a href="https://github.com/talonx/python-scribbles/blob/master/fb/breathalyzer2.py" target="_blank">second version</a> favours a smaller word whenever the scores for two are equal.<br />And the result is (Score remains the same)<br /><blockquote><i>TIS SENTENCES IS NOT VARY GOD</i></blockquote></div>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-513893015191696787.post-76108147882751414102011-04-03T05:29:00.000+05:302012-06-17T14:07:00.023+05:30A study planI have been formulating a self-study plan for some time now. The plan involves mostly math subjects, as I finally want it to culminate in data analysis/mining/AI/ML - which again are subjects I've always been interested in but never took up seriously. Looking at the experience of people who have done such a thing before, a solid foundation in the underlying math is absolutely essential. So, I've started off with statistics and probability. Having been out of touch for a long time (15 years!) with the basics of either of these, I chose to begin simply - Sheldon M Ross's Introductory Statistics supplemented with <a href="http://www.khanacademy.org">Khan Academy</a>'s video lectures (These are perfect for brushing the dust from those rusty concepts). After finishing this I'll aim for a book like <a href="http://www.amazon.com/Introduction-Probability-Dimitri-P-Bertsekas/dp/188652940X">Bertsekas</a> combined with MIT's OCW lectures. <br/><br/>Have decided to put the solutions to the exercises I'm doing <a href="http://statsprob.blogspot.com/">here</a>.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-48234155488952290232011-01-30T05:07:00.000+05:302012-06-17T14:12:42.354+05:30gcc does not check out of scope names in unreachable code<div dir="ltr" style="text-align: left;" trbidi="on">While attempting to write a small HTTP server in C, I copied some code over from a previously written C file and immediately noticed a bug.<br /><br /><i>File httpd.c</i><br />#include "../mynet.h"<br /><br /><div style="font-family: "Courier New",Courier,monospace;">if(errno = EINTR) {</div><div style="font-family: "Courier New",Courier,monospace;"> //do something</div><div style="font-family: "Courier New",Courier,monospace;">} else {</div><div style="font-family: "Courier New",Courier,monospace;"> err_sys("read error")</div><div style="font-family: "Courier New",Courier,monospace;">}</div><br /><br />Yes it's a stupid beginner mistake - typing the assignment operator instead of the equals check. The thread of execution would never enter the else block. I corrected it, but the interesting part came when I tried to compile it.<br /><br /><div style="font-family: "Courier New",Courier,monospace;">cc ../mynet.c httpd.c</div><br /><br />mynet.c contains some handy helper functions that I've used in my other server classes. Guess what - the compilation failed with this message<br /><br /><br /><div style="font-family: "Courier New",Courier,monospace;">"httpd.c:(.text+0x6a): undefined reference to `err_sys'"</div><br /><br />I checked my header and the err_sys function was nowhere to be seen. If this function is missing, how did my other class (from where I copied this code) compile previously?<br />After some fiddling around I put the assignment operator bug back, and guess what? The code compiled fine. <br /><br />Based on just these observations, we can conclude that the gcc C compiler ignored the unreachable (else) part of the code. It did not even check if the code inside the else block was legitimate. How far did this behaviour go? Let's see.<br /><br /><i>File httpd.c</i><br /><div style="font-family: "Courier New",Courier,monospace;">#include "../mynet.h"</div><div style="font-family: "Courier New",Courier,monospace;"><br /></div><div style="font-family: "Courier New",Courier,monospace;">if(errno = EINTR) {</div><div style="font-family: "Courier New",Courier,monospace;"> //do something</div><div style="font-family: "Courier New",Courier,monospace;">} else {</div><div style="font-family: "Courier New",Courier,monospace;"> mocha(); //Undefined function</div><div style="font-family: "Courier New",Courier,monospace;">}</div><br /><br />This compiles fine.<br /><br /><i>File httpd.c</i><br /><div style="font-family: "Courier New",Courier,monospace;">#include "../mynet.h"</div><div style="font-family: "Courier New",Courier,monospace;"><br /></div><div style="font-family: "Courier New",Courier,monospace;">if(errno = EINTR) {</div><div style="font-family: "Courier New",Courier,monospace;"> //do something</div><div style="font-family: "Courier New",Courier,monospace;">} else {</div><div style="font-family: "Courier New",Courier,monospace;"> asdf;</div><div style="font-family: "Courier New",Courier,monospace;">}</div><br /><br />This correctly fails with an error.<br /><br />So syntax checks are being done in code that is known to be unreachable, but there are no checks for undefined functions. A bug? I would say yes. Google did not turn up much except this old link - <a href="http://compgroups.net/comp.lang.c++.moderated/could-if-else-avoid-syntax-checking-compile-time-unreachable-code" target="_blank">http://compgroups.net/comp.lang.c++.moderated/could-if-else-avoid-syntax-checking-compile-time-unreachable-code</a></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-513893015191696787.post-38545258541193149292011-01-09T17:45:00.000+05:302012-06-17T14:06:59.973+05:30Working through UNPAs I wrote <a href="http://code.deepinspace.net/2010/11/23/falling-through/" target="_blank">previously</a>, I've been working through Richard Stevens' Unix Network Programming (3rd Ed) Vol 1. It covers the basics of the Sockets API in UNIX and similar OSs.<br/><br/>Unfortunately I've been able to devote time for this mostly on the weekends. This translates to slow progress because UNP goes into a lot of depth about everything. This is a good thing, but it also means that I've to reread and review the last few pages everytime I try to pick up where I left off. This does not really help when I'm trying to understand concepts in depth. So what's the solution? I'll try to do a bit atleast 3-4 days a week from now on.<br/><br/>UNP is an amazingly detailed book - and as one of my colleagues said - "If you read that book properly, Stevens makes sure that there's nothing left for you to know on the topic". I agree.<br/><br/>Stevens wrote his own wrapper functions over common socket functions and used them in all the code examples in the book. The wrappers handle all error codes and portability issues (like IPv4/IPv6). These are included in a header unp.h (available in the back of the book as well as online on <a href="http://www.unpbook.com/src.html">http://www.unpbook.com/src.html</a>).<br/><br/>Some reviewers of the book gripe about this and say that this is an obstacle to learning the actual functions. But I think that there was no other way to do it without littering every example snippet with code for portability and error handling. The wrapper strategy makes it easier to follow the examples, and at the same time - as I found out - it makes you write those wrappers yourself. True, you can just include the unp.h header as you try the examples, but then you'll never know what those functions are doing. I've found that creating my own header and writing the functions as I come across them, after looking at the book's source code, works great. Most of them will end up identical to those in unp.h.<br/><br/>I'm pushing the examples I'm trying out into <a href="https://github.com/talonx/C-Scribbles/tree/master/net">github</a> - it's a scratchpad so not everything might compile.<br/><br/>I've added a generic <a href="https://github.com/talonx/C-Scribbles/blob/master/net/mynet.h">startserver function</a> to my header - this takes a pointer to a function as an argument. The generic function starts a server socket (bind/listen/accept), forks a child when a client connects and calls the function that was earlier passed as an argument, abstracting out the actual serving part. The function pointer syntax was not hard to figure out - I'd read <a href="http://www.amazon.com/Expert-Programming-Peter-van-Linden/dp/0131774298">Peter van der Linden</a>'s algo on unscrambling declarations in C last week. Interesting how things add up!Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-513893015191696787.post-28113572002192116972010-10-08T02:35:00.000+05:302012-06-17T14:13:07.053+05:30Everybody's Recommending<div dir="ltr" style="text-align: left;" trbidi="on">Is Google the only one who has possibly accumulated a lot of data on your online activities?<br /><br />Think again.<br /><br />Most of us use one of these -<br /><ol><li>Facebook<br /> </li><li>Twitter<br /> </li><li>ShareThis<br /> </li><li>Technorati/Digg/et al</li></ol><br />There's a common aspect to all these networks/tools - all of them can potentially collect data about the online preferences of their users. So - do they? Some of them do.<br /><br />Online preferences are links that you visit, which translates to things that you are interested in. This kind of data can be used to build up a profile of the user.<br /><br />Think about it -<br /><br />1. Facebook knows what you share on facebook.com, knows what you "Like" among others' shared links, and now with OpenGraph knows what you "Like" on sites that have the Facebook Like button.<br /><br />2. Twitter knows what links you share, and now with t.co - its own shortening service - it will know what links shared by others you clicked on (read "interested in"). From a <a href="http://blog.twitter.com/2010/06/links-and-twitter-length-shouldnt.html" target="_blank" title="http://blog.twitter.com/2010/06/links-and-twitter-length-shouldnt.html">Twitter blog post</a> -<br /><div style="padding-left: 30px;">routing links through this service will eventually contribute to the metrics behind our Promoted Tweets platform and provide an important quality signal for our Resonance algorithm—the way we determine if a Tweet is relevant and interesting to users</div><br />3. ShareThis - if you're logged into ShareThis, it knows what you shared.<br /><br />Links you share and visit provide a picture, albeit incomplete, of your online preferences.<br />The question is, how are these tools and services planning to use this data?<br /><br />If you know what someone likes, you can recommend stuff to that person. A lot of sites do this already. These recommendations are based on multiple parameters. E.g. <a href="http://www.cs.umd.edu/%7Esamir/498/Amazon-Recommendations.pdf" target="_blank" title="http://www.cs.umd.edu/~samir/498/Amazon-Recommendations.pdf">Amazon's recommendation system</a> - which does a great job - uses collaborative filtering. Simply put, it uses data from your past purchases, ratings and I-Own-This history and from other users whose history is similar to yours. The more history you have on Amazon, the better your recommendations get.<br /><br />Building a content recommendation system seems to be an obvious step once you have a data mountain of your users' likes. And this is what these sites seem to be doing but to achieve different ends.<br />E.g. Facebook - See slide #29 <a href="http://www.slideshare.net/CMSummit/ms-internet-trends060710final" target="_blank" title="http://www.slideshare.net/CMSummit/ms-internet-trends060710final">http://www.slideshare.net/CMSummit/ms-internet-trends060710final</a>. This has not happened yet, but what's stopping it, considering what <a href="http://www.readwriteweb.com/archives/facebooks_zuckerberg_says_the_age_of_privacy_is_ov.php" target="_blank" title="http://www.readwriteweb.com/archives/facebooks_zuckerberg_says_the_age_of_privacy_is_ov.php">Mark Zuckerberg said earlier this year</a> ?<br /><br />Twitter has recommendation plans - <a href="http://groups.google.com/group/twitter-development-talk/browse_thread/thread/14d5474c13ed84aa?pli=1" target="_blank" title="http://groups.google.com/group/twitter-development-talk/browse_thread/thread/14d5474c13ed84aa?pli=1">http://groups.google.com/group/twitter-development-talk/browse_thread/thread/14d5474c13ed84aa?pli=1</a><br /><br />ShareThis <a href="http://blog.sharethis.com/2010/04/20/influencers-engage-mederma/" target="_blank" title="http://blog.sharethis.com/2010/04/20/influencers-engage-mederma/">already has behavioural advertising in the works</a> with its segmentation technology.<br /><br />The bottom line is - some of these services are going to use it to improve the end user's experience - and will do so within the boundaries of their privacy policies. The rest - we don't know.</div>Unknownnoreply@blogger.com0