Mohammed Naviwala

Maximum TCP Throughput

2012-06-18T23:33:00.002-07:00

So let's go through a scenario. I have 2 DCs (in 2 different cities) and have 1 Gig link between them. However, max throughput I get between 2 servers is nowhere near my link capacity. So what could be the reason?

This is how the TCP throughput is calculated.

TCP Window size (in bits) / Latency in seconds = TCP Throughput.

So in above example, if my TCP windows size is standard 64K and assumes the latency between 2 DCs is 20 milliseconds, then the maximum throughput I will get is

524288 / .02 = 26214400 that is 26.21 Mbps

Now as we can see, 2 factors that decide the maximum throughput on a link are

· TCP window size

· Latency in seconds

Now, what can we do to maximize the throughput? One such solution is to increase the TCP window size on the servers. So in my case, what should be the maximum window size on my servers to get the 1Gig throughput? Let's calculate

X / 0.02 = 1Gbps, that is 2500000 bytes or 2441.4 KBs

But then there are issues with increasing the TCP window size like the retransmission of the whole window is needed if a packet is dropped and it also requires deep buffered on servers until the acknowledgement is done from the other side.

For solving the above issues, we can have a WAN optimization device at the edge. This device can do the TCP selective retransmission and also the local ack to the server. They also do layer 7 optimization which can increase the link utilization.

MSS Clamping

2012-06-12T05:04:00.003-07:00

I was reading through my past documents and came across a scenario where I had to reduce the TCP MSS on routers for the connection to work propoerly. So I am writing this post in order to note it down and to help anyone looking for similar solution.

We were implementing DMVPN network and during the migration, the network was very slow. The client did not accept the test and we had to rollback the migration. Then we set up a small POC mirroring the client environment (we did a POC before migration but did not reflect the specific client network) and started troubleshooting and soon we realized that we had to reduce the TCP MSS (Maximu segment size) on the router for DMVPN to work properly.

We faced similar issue with another client this time with GET VPN implementation and the same solution did the trick for us. This act of reducing the TCP MSS is called MSS Clamping. On cisco routers, we can use following commnds to reduce the TCP MSS.

interface gi0/1

ip address 10.10.10.1 255.255.255.0

ip tcp adjust-mss 1360

Unicast flooding - Follow up

2012-06-12T03:04:00.003-07:00

Unicast Flooding with Asymmetric routing

As a follow up post to the unicast flooding, lets take the below scenario.(Diagram taken from Cisco website). Here, Server 1 (S1) is in Vlan 1 and server 2 (S2) is in vlan 2 connected to 2 different switches. S1 is connected to SW1 on int f1/1 and S2 is connected to SW2 on int f1/1. These 2 switches are connected to each other over a .1Q trunk carrying both the vlans on int f1/24 on both switches. Router A and Router B are doing routing for both the vlans (Router-on-a-stick). Router A is connected to SW1 on int f1/23 and router B is connected to SW2 on int f1/23. Router A is the gateway for Server S1 and Router B is the gateway for the Server S2.

Now when there is traffic passing from the S1 to S2, following path will be used

S1(Vlan 1) – SW1 – Router A – Vlan 2 – SW2 – S2 (Vlan 2)

The reverse path will be

S2 (Vlan 2) – SW2 – Router B – Vlan 1 – SW1 – S1 (Vlan 1)

Let us analyze why there will be unicast flooding each time a packet travels between S1 – S2

Scenario 1: Traffic traverses from S1 to S2

Here, S1 knows that the destination address is not in the subnet and hence needs to ARP for its own gateway (Router A Vlan 1). It will send the packet to the Router A vlan 1. Router A in-turn needs to send the packet to S2 which is in the network connected to its Vlan 2 interface. So it will use the Vlan2 source MAC and will ARP for the MAC of S2. This Arp reply will come from the trunk port. Router A will send the packet directly to S2. Now this packet will be sent out interface 1/23 on SW1 but since there is no MAC entry on SW1, this packet will flooded to ach port except the source port. SW2 will receive this packet on the trunk port and learn the MAC of Router A vlan 2 interface and store it in the CAM against the trunk port. However, it does not have the CAM entry for S2 and hence it will also resort to unicast flooding. So the following MAC learning has been done yet

On SW1

For the S1 MAC address on interface 1/1

For Router A vlan 2 Mac on interface 1/23

On SW2

For the Router A vlan 2 Mac on interface 1/24

Scenario 2 : Return traffic from S2 to s1

On the return side, S2 needs to send the traffic to S1 which is not in its subnet hence S2 will ARP for its own gateway which is Router B Vlan 2 and send the traffic to it. Router B will receive the traffic and realizing that there is a route, will ARP for the S1 which it will receive on VLAN 1 and send the packet to S1. This packet will be sent on interface 1/23 on SW2. SW2 will learn the Router B Vlan 1 MAC on interface 1/23 however it does not have MAC entry for the S1 and hence SW2 will resort to unicast flooding. This packet will be received on SW1 on interface 1/24 and SW1 will learn the MAC of Router B Vlan 1 on this interface. This time around, the SW1 has the CAM entry for S1 and hence this will be unicast and not flooded. during this phase, following MAC learning has been done.

On SW2

For the Router B Vlan 1 on interface f1/23

For the S2 on Vlan 2 on interface f1/1

On SW1

For Router B vlan 1 on interface f1/24

As we can see, on SW1, we never learn MAC for S2 and hence every forward packet will be unicast flooded into Vlan 2. Similarly, on SW2, S1 MAC is never learnt hence every return packet will be unicast flooded into Vlan 1.

As always, I can be wrong so any corrections/suggestions welcome.

Why 53 bytes for the ATM cell?

2012-06-11T05:26:00.001-07:00

Today I will post an interesting and somewhat shocking reason why the cell size of the ATM network was fixed at 53 bytes. The choice of 48 bytes was a political matter. When the ATM standard was being ratified, parties from the United States wanted a 64-byte payload because this was felt to be a good compromise between larger payloads optimized for data transmission and shorter payloads optimized for real-time applications like voice; parties from Europe wanted 32-byte payloads because the small size simplify voice applications with respect to echo cancellation. Most of the European parties eventually came around to the arguments made by the Americans, but France and a few others held out for a shorter cell length. With 32 bytes, France would have been able to implement an ATM-based voice network with calls from one end of France to the other requiring no echo cancellations. 48 bytes ((64+32)/2) was chosen as a compromise between the two sides. 5 bytes of header was also provisioned assuming that the 10% of the total payload was the fair price to pay for the routing info. hence 48+5 = 53 bytes size was standardized for the ATM cell.

Cisco switches and Microsoft NLB

2012-06-04T03:56:00.002-07:00

Microsoft NLB uses 2 modes

1. Unicast - where the actual MAC address of each server in the cluster is replaced by a common NLB Unicast MAC address. This is ok if all the servers in the NLB Cluster are connected to different switches. however, when all the servers are connected to the same switch this does not work. to solve this issue, MS has done a work around whereby a bogus MAC address per server in the cluster is created and that is assigned to the server. However the difference being this Bogus MAC is used in the Ethernet frame and not the ARP Replies. For the ARP replies to the clients, the common NLB MAC is used. Hence switch does not have an entry into the CAM table for this common NLB MAC and hence has to resort to flooding. Solution here is to have a hub in front of the NLB server cluster.

2. Multicast - In this mode, the cluster members respond to the ARP replies by sending the Cluster Multicast MAC Address (Which is illegal btw). The problem here is that the cisco switches do not accept the ARP replies where the requested IP Address is unicast but the replied MAC is multicast. hence we use the static mac entries to populate the CAM tables of the switches where the NLB cluster servers are connected.

IOS commands to add these entries are

mac-address-table static 0300.5f11.0011 vlan 10 interface fa1/1 fa1/2

Unicast Flooding

2012-04-21T15:50:00.001-07:00

LAN switches will use the CAM tables to forward packets. and these tables are filled by a process called MAC learning whereby switch notes down the MAC address from the source field of the frames and tie the interface to that MAC. but what happens when the switch does not know (or has not learnt yet) the destination for a particular MAC? it will forward it to all the interfaces except the one it received it from. This process is called unicast flooding.

Most of the switches do it per VLAN and this coupled with asymmetric routing can create a condition where by switch cannot learn the destination for specific traffic and has to resort to unicast flooding everytime it has to forward a packet. I will take that in a follow up post.

Global Financial Turmoil

2008-09-25T14:16:00.000-07:00

Everything was appeared good. until one day Lehman Brothers announced bankruptcy followed by the mighty buy-out of Morgan stanley by the BOA and the hell broke out. dont you think these kinds of periodic 'bubble bursts' are a healthy sign for the global economy? Well I, certainly do. If we analise the root cause of this turmoil, we get to know that the indiscriminate spending on the part of few individuals(read Americans) and indiscriminate lending on the part of few investment banks(read American again) has led to this disaster which is now threatening the entire world economy.

Lets look at how this all started. Indiscriminate credit-lending even to those with shabby credit scores started this crisis. Fall in the greenbacks value and corresponding fall in the US property market fueled it. people couls no longer afford paying back of the credit they had obtained and hence started defaulting. This put immense pressure on the banks and pushed them into the credit crunch. they could not sustain and eventually had to default or had to sell their stakes. it started with CITI, followed by Bear sterns and reached upto LB and MS.
Lets llok at the solutions.

One such solution mooted by the US governent is he mammoth US $700 Bn bail-out package. can this save the banks? yes it can. but the question is can this restore the investor confidence??? well no it can do so in immediate future. It can rather provide much needed relief to the traped investors that their investment stands a chance to returned back in principal not if the returns on it.

But this turmoil, as I belive, would rationalise those individual at mistake of over-spending their limits and also those who tend to learn from others mistakes that they canot sustain their irraltional behaviour forever. They, eventually, will have to come to task by selling whatever they have and to pay the credit even if it means they are rendered homeless. This, on the other part, will also serve as a very serious warning to banks that gravely miscalculated credit-lending can lend them nowhere but have themselves blown up.

That the 158 year old institution which can stand 2 world wars and disastrous recession cannot withstand the credit crunch says all abt the massive scale of the problem. The solution, if any, should also needs to be of the same proportion. Hope this gets sorted out at the earliest and does not thretens the on-slaught of global recession.

A story of the cracked pot

2008-08-21T02:39:00.000-07:00

I have ever wondered why people specially in our part of world have loads of advices to give. probably they think they are the best ones and the perfect example of the mr. perfectionist. Anyways I was thinking about the shortcomings of me with comparision to few of the people I know. Many streams of vibrant but negative ideas kept my mind busy. This process, although slow, continued for a number of days. I can say I was in that paranoid situation where I compared each of my actions against a bunch of people and for my every action, surprisingly, I found that I had a shortcoming.

Just as my mind was about to resolve me as the most useless fellow on this beautiful creation of the almighty which we know by the name earth, this little but a nice story of a cracked pot happened to present itself in front of my eyes.

The story is of a water bearer who had 2 pots each hung on the ends of a pole which he carried on his shoulder. one of the pot had a crack in it while other was a perfectly nice pot. At the end of the long walk from the water stream to the house, the cracked pot would only arrive half. the other one would be full.

Of course the perfect pot was proud of its achievement and would look down on the cracked pot and the cracked one, just as in the defensive mind as me, would be very sad that he had a crack and could not deliver to the fullest capacity. This continued for a couple of years. one day, feeling it unbearable, the cracked pot told his master that it felt ashamed of his shortcoming and his master had to suffer on account of his shortcomings.

Hearing this the water-bearer asked to the pot that did he notice the flowers on his side of the pole? he said to the pot that he planted the flower seeds on his side of the pole so that on his way back to house, he can water the flowers. The water-bearer told the cracked pot that if it were not cracked, he could not have got the flowers which he uses daily to beautify his house.
So just as in this case, where the perfect pot was proud of his delivery to the fullest capacity but could not help other just for the reason that it was left with no spare capacity to help, I thaught that the people I compared myself to had accomplished few things but had nothing more to achieve else deliver on their so called perfection.

But for the people like me, not so perfect not so fullest, we are all like the cracked pots. Its the cracks in us which makes our life so very interesting and rewarding. you have got to take each person as they are and to look at only good in them. For being perfect means god but it also implies being bent into a shape left with no flexibility. As said

"Blessed are the flexible, for they shall not be bent out of shape"

God Bless.

Believe in Yourself

2008-08-01T15:37:00.000-07:00

I believe that I was lucky to have suffered. Some people don’t realize that in suffering there is great potential, because if you are deprived for any reason.. politically, financially, socially or otherwise.. and if you set your mind in the right direction, you will find that the only way to survive is for you to excel, by being better so you can be treated better. The turning point, I think, was when I really realized that you can do it yourself. That you have to believe in you because sometimes that's the only person that does believe in your success but you.

I was told a lot about being content in what you have. A plethora of people advised me that one cant get more than his fate. My question to them was 'has anybody seen his or my fate?'. The answers were obvious. But I had faith in God, belief within and courage to dream. As quoted in the holy Quran 'God Tasketh not a soul beyond its scope'. Believe that you have to go through this because you have the capability to do so. Believe that you are the one chosen by god to be a proof his principles to the rest of humanity. Accept it as a divine mission assigned to you by the almighty.

If you believe that some day it's going to happen, some day it probably will happen. You just have to make sure you're there when it's happening, and ideally you're at the front of the parade, and the principle beneficiary of when it happens, but it's not a kind of thing where you just sort of sit back and wait.

The most important thing is God's blessing and if you believe in God and you believe in yourself, you have nothing to worry about.

God Bless.