Linuxdragster

Cloud Native lingo

2019-06-23T00:11:00.000-07:00

This is my understanding of the various terms/names, used in the context of Cloud Native Infrastructure or deployment. Trying to put my thoughts into words, so it might help others or maybe others would disagree me via comments, and that would help me correct myself. Well... at least, that is the thought.

The documentation would be heavily skewed towards Linux, since that's what I normally work with.

What are Containers?

In simple terms, it's a collection of processes on the host, isolated to run with its own set of resources. Linux namespaces and Control groups, helps a user to either carve out/create pseudo resources(with the help of other kernel drivers like tty, devpts ...) and enforce limits on shared resource consumption.

Namespaces, a kernel feature like PID, UTS, NET, IPC, User & Mount can be controlled at process granularity to selectively control the required isolation. But normally for containers, all namespaces would be enforced, since the processes running within the container should be isolated completely from the host.

Control group or cgroups(usually referred) are a kernel feature, that help the namespaced processes(defined above) to be limited in its resource consumption of shared resources, like CPU, Memory, Disk I/O and Network I/O.

In short, containers are a bunch of processes, isolated by namespaces and resources are limited by cgroups.

Docker

It's the name of the company, as well as the software(Container Orchestrator) that it provides, to help build & deploy containers in an efficient and easy manner. Docker relies on underlying Linux namespaces and cgroups to provide the container(isolation & resource limitation) functionality. dockerd, along with its cohorts from OCI specification (containerd, runc and libcontainer) provide the lifecycle management of the container, as well as manages the image push/pull from remote Docker registry to local registry.

Before Docker, Libvirt's LXC driver was used to deploy containers. But building the container images was not structured and each team would have to develop their own tools. This was a big pain point, especially in a development environment, when the bits used in the container, used to change on a daily basis. Docker came along and made it easier to build container images, as well as to distribute them using their Registry format. This made a big difference for developers and everybody hooked on to this mechanism of deploying containers and made containers to what they are today. A first class citizens, in all Public Cloud offering.

Docker CE(Community Edition) is the free version and Docker EE(Enterprise Edition) is the paid version with support from Docker.

Kubernetes

Kubernetes or K8s is a Container Orchestration Engine(COE). The project to develop this software, originated from Google(heavily influenced by Google's internal project Borg) and later open sourced. Kubernetes cannot deploy containers on its own. It relies on Docker/CRI-O or any other Container Orchestrator that implements the OCI specifications(Runtime & Image) to deploy the containers.

To help make the differentiation between Docker & K8. Docker can be used to deploy containers on a single node. If you have multiple nodes that can be used to deploy containers, then you use K8s as a scheduler, that can talks to these individual Docker instances and deploy the container on suitable node based on the resource requirements of that container.

In short, it's a layer on top of Docker to help deploy containers on a multiple node setup called K8s cluster. This is especially useful in cases, if one of the node becomes unhealthy due to resource shortage or hardware issues, K8s would be able to migrate the existing containers to an another node in the cluster. Also since K8s scheduler would have an eagle's eye view of all the available/occupied resources, it would be able to distribute the deployment of containers onto the various nodes evenly.

Furthermore, the smallest deployable unit in K8s is called a POD. It's a collection of containers, allowing the containers to share network and filesystem namespace. This is designed, based on the real world needs to run a collection of containers, as a single unit for execution(share resources for communication like shared memory..._), but still keep the containers independent from packaging perspective. It could very well be written in a different language and it would not make a difference.

You can compare K8s with Docker swarm, but K8s has pretty much won the public vote, as a Cloud Native Container Orchestrator Engine. K8s does provide additional functionality and is not limited to, network connectivity using CNI(Container Network Interface) plugins, persistent storage using CSI(Container Storage Interface) plugins and vendor specific device plugins(to support GPUs and other vendor specific devices). These features, along with its robust community support, makes it a first choice for both on-premise, as well as public cloud container orchestration.

CNCF

Cloud Native Computing Foundation, a vendor neutral committee responsible for governing the various software components that help with deploying/maintaining/monitoring software in Cloud Native Infrastructure. Micro Services is one of the main software architecture that is publicized and supported by the CNCF community.

OCI

Open Container Initiative is governed by CNCF and it includes the Runtime specification and Image specification for container deployment. The goal is to ensure multiple disparate components that deal with cloud native infrastructure would be able to work together, as along as they stick to the specification for interface compatibility. containerd, runc and libcontainer are few reference implementations of these specifications.

Watch out for other topics like Service Mesh, Micro Services, K8 plugins in later posts.

References:

[1] containerd

[2] Libvirt LXC

[3] OCI

Port Knocking

2010-03-04T04:08:00.000-08:00

So, when you were kids, you might have had a secret knock sequence like "tap taptap tap" to open the club doors, so that only the kids that you like and who knew sequence could enter the club. Now port knocking is about trying to access a few ports in a specific sequence which could alert the firewall to execute some action on your desktop/server. It could be a shell script which monitors your iptables log file and executes a specific action on parsing a specific sequence of ports.

Some of the possible action that I can think of
(1) Open up ssh port on your Linux desktop so that you can access it remotely and
close it once you are done.
(2) start a backup of your hard drive.
(3) If you want to let some one inside your network without others noticing... :P
(3) ... and lot of other stuff shady also...

Seems interesting...
check out more here.
http://www.itworld.com/it-managementstrategy/98567/port-knocking

HTTP/FTP as seeds for Bittorrent Client

2010-01-02T17:15:00.000-08:00

Recently, I came across a fact that some of the P2P clients like BitTorrent and Vuze support HTTP/FTP also as part of their content source. That is really cool, cause it gives us an advantage of having a peer which does not upload data based on Tit-for-Tat, and if they are running an HTTP or FTP process, most probably their uptime will be higher compared to other Peers in the swarm.

Yeah, but don't be too happy, cause the server side involves a script to avoid overloading the web server process as a precautionary measure for malicious attacks.So you can't just be requesting content continuously, but carefully timed request can help us to download content without actually uploading any data to any other peer.

I do not know ,as to how many users actually host via P2P & HTTP/FTP simultaneously .May be corporations like Ubuntu, VMWare may support this ,since they can afford the servers.

The metadata key is 'url-list' for the .torrent file.

For the TechSavy check out the below links for detailed info and specs:
[1] http://www.getright.com/seedtorrent.html
[2] http://www.bittornado.com/docs/webseed-spec.txt

DNS with Geo location....

2009-12-15T11:13:00.000-08:00

Did u know that DNS has this feature with BIND version 4 and greater, that supports LOC records? Well Records are information that you can normally get from a machine that runs a DNS server Process. LOC Record is a specific type that provides information regarding the latitude,longitude & altitude of the server. Basically helps you to locate the DNS server on a geographical map, thereby identifying location of the machines, which use that DNS server as the Authoritative server. Pretty cool, if you are trying to map the clients that you are interacting with on a the map.

But the disadvantage of this mechanism ,is that malicious users can identify the physical location ( @least narrow it down) using this method, thereby causing security concerns... Due to this reason, most of the DNS server's do not support this specific request type or provide incorrect information. This sucks!! :(

Well... due to this, I need to find different ways to identify the location of the clients that I interact with for my project... That again sucks!!! :(

Share files using Torrents....

2009-11-19T17:31:00.000-08:00

We are going to read through a few set of simple steps ,that could help you to share huge content with your fnds and family using P2P. As soon as I said P2P, the first thing that pop2 to your mind is pirated content. Well P2P is used to share a lot of Pirated content, but not all content is pirated. It even used legitimately for sharing big files such as Fedora, Ubuntu OS Images, Free S/W, other content... etc :)))

Anyways

1) Create a torrent file, that basically creates a snapshot of u r file in few words.This is used for verification and also contains the tracker URL.

2) Share the torrent file with other users who would like to download this content.

3) The public tracker that I use is "http://tracker.openbittorrent.com" and this information is provided in step 1.

Once u r fnds use the torrent file using any of the torrent clients like Vuze, Transmission, the client would contact the public tracker, which would then provide your IP address* for download.

More the # of users, the less the amount of time it takes for the download.

*NOTE: You need to load u r torrent to be hosted ,else the tracker would not know that you have the content to be shared.

Hope this is simple to understand...

-Alphy

Basics of Peer to Peer

2009-11-01T01:06:00.001-07:00

Many are using P2P application to download both legal & Illegal content from the Internet ( Ya, I know you don't do it...:P ).Vuze formerly known as Azureus is popular since it supports torrents in addition it allows the user to search for content using keywords which is not yet supported by any other Bittorrent client. Furthermore it even has a graphical view of the files pieces being downloaded, the distance between the various clients with whom you are connected and other stuff which provide easy to understand info to the naive user.

well some basic terminology first

1) Peer : Any computer on the Internet that runs a Bittorrent protool supported application.
2) Seed : Peer's who have completed the download of a specific file and are online, just to upload to other peers interested in the same file.
3) Leech: Peer's who are actively downloading the content.
4) Swarm: Peer's connected to each other actively swapping file blocks form a swarm. In Basics terms, each file will form its own swarm for distribution.
5) Freeriders: Peers,who download content but never upload content to other peers when requested.
6) Tit-for-Tat: This is a transfer mechanism implemented in Bittorrent protocol to avoid freeriders.For each forward block transfer, they should be a transfer in the reverse direction, else the connection is discontinued.
7) Tracker: Responsible for hosting the torrent file that contains info regarding the file size, HASH for validation and also the coordinator for various peers.
8) # of connection: 5 active connections & 30 passive connection.

well this is the stuff that I know, update this list ASAP with new info.

-Alphy

Content-Centric Networks

2009-02-28T07:23:00.000-08:00

I recently viewed a video article by Van Jacobson on Content Centric Networks.The concept is really awesome and just what the INTERNET needs now to improve its usability.Well Let me explain what this fuss is all about!

To access any specific content,you have to access a specific site that contain the content for retrieval.So you need to know the location of the server that contains the information that you need.Well this made sense in the past before the days of google or yahoo's search engines.Now it is possible for us to know the location of the days by just typing the name/tag of the content that we need.The search engine does the job for and provides the source location of this content.

Now consider this concept in which the content is not stored in one place but it could be stored on your shared disk,which you,out of u r big heart have donated to the INTERNET. Now you have multiple source locations of the same content. Being said as the content being literally on the INTERNET cloud.

This helps us to avoid a central server being swarmed for a popular content and localize the content delivery.Helps us to avoid unnecessary network traffic.

well does it not look like a P2P system or even caching for that matter.I did think it was like that but P2P works well in case of popular content,cos then you have more seeders or in case of caching there is still a central source. Remove the term popular content and make it possible for all content and make is available everywhere (users shared disk driver )instead of specific cache servers and there you go,you have content centric networks.

Linux Netfilter Hooks

2008-10-03T20:36:00.000-07:00

Recently learned about the Netfilter options provided by the Linux kernel,so I though sharing it here might help me to understand better...

So Let's c Netfilter is a mechanism by which Linux allows us to capture packets both inbound and outbound from the system.It has has two major hooks,called PRE-ROUTING and POST-ROUTING.

All packets inbound to the machine can be caught at the PRE-ROUTING phase and the packets outbound from the machine can be captured at the POST-ROUTING phase.As the filter name suggests they are caught before and after the routing decision has been made based on the information on the routing table.

These are called as hooks where u can register your function that needs to be executed based on the conditions.

Packet information will be provided in structure called skbuff.This has a basic set of functions that can be used to trim/append data to the packets and resend them onto the link layer or to the application.

Well this is the basic overview....

until I get more info...

signing out..

-Alphy

Are Desktops ,the future code breakers....

2007-02-13T03:03:00.000-08:00

Teraflops,the latest invention from Intel.Don't get in to an frenzy,they ain't for commercial use yet....

Teraflops,have 80 core's ,which is like exponential compared to the four cores,found on desktops.Each core is capable of performing their own processing.That means now,we are limited only by the capability of the Software ,and not the hardware to perfom multiprocessing.

Of course missile silos will use this technology,but technology becomes really useful,once every tom,dick and harry can lay their hands on them.... so where does this leads us to ?

abosulte bliss?

Code breakers and Hackers must be really happy with this new inventions,cos this just simpfies their desire "The need for processing speed".

the ultimate breakthrough is the size of this processor,it is not bigger than a fingernail :) .swap u r mobile for this and u would be performing astronomical calculations in nanoseconds!!

I wish that would be the connection of my internet,then life would be really easy for me!!

-Alphy

Subversion -version control tool

2007-02-07T23:15:00.000-08:00

This is a new version control tool developed by "collabnet" and it is cool,cos this is an open source software like CVS,excluding all the drawbacks of using the same tool.

For instance Subversion supports, directory versioning.That helps us a lot from CVS.Even though i am not an user of CVS,i du know some of its drawbacks :) .subversion has implemented a virtual filesystem similar to MVFS in Clearcase,which helps it to maintain and display information regarding the directory versions....

Even though it is an open source tool,mostly users from CVS would move to their similar arena.BUt would users replying on clearcase ,would they move to Subversion?seems like only time can answer this question :)

-Alphy

Christening

2006-12-22T01:04:00.000-08:00

Hi,

I have just thought about creating this new technical blog,to post my ideas on Linux and it evolution in the server and desktop market.here you might find stuff related to clearcase,even java programming.but i christened the blog as linuxdragster,cos i am linux buff.

Any way look forward for real technical update reagarding anything :)

check out my other blog http://www.mymixedreactions.blogspot.com for posts on a generic,lighter note of view on life .

-Alphy