By Admin on

What is TCP incast problem?

Incast is a many-to-one communication pattern commonly found in cloud data centers implementing scale out distributed storage and computing frameworks such as Hadoop, MapReduce, HDFS, Cassandra, etc. — powering applications such as web search, maps, social networks, data warehousing and analytics. Incast can also more specifically be referred to as TCP Incast, as the cloud applications creating this communication pattern rely heavily on TCP.

The basic pattern of Incast begins when a singular Parent server places a request for data to a cluster of Nodes which all receive the request simultaneously. The cluster of Nodes may in turn all synchronously respond to the singular Parent. The result is a micro burst of many machines simultaneously sending TCP data streams to one machine (many to one).

The Incast traffic can be very short lived flows (micro burts), depending on the application.  For example, you could have a Parent server request 80KB of data across 40 Nodes.  Each Node simultaneously responds with 2KB of data.  That’s just two packets from each Node.  In a real world scenario, the Parent could be a database server requesting a 80KB photo of the newest friend added to your social network.

“TCP Incast” may also be used in a context referring to the detrimental effect on TCP throughput caused by network congestion the Incast communication patterns create. This simultaneous many-to-one burst can cause egress congestion at the network port attached to the Parent server, overwhelming the port egress buffer. The resulting packet loss requires Nodes to detect the loss (missing ACKs), re-send data (after RTO), and slowly ramp up throughput per standard TCP behavior. The application issuing the original request might wait until all data has been received before the job can complete, or just decide to return partial results after a certain delay threshold is exceeded. Either way, the speed, quality, and consistency of performance suffers.

The congestion caused by Incast has the effect of increasing the latency observed by the application and its users. There’s a give and take with application performance as Node cluster sizes grow. Larger clusters provide more distributed processing capacity allowing more jobs to complete faster. However larger clusters also mean a wider fan-in source for Incast traffic, increasing network congestion and lowering TCP throughput per Node.

SPIRO Google Plus