stressed out

Shounak Ray, Jacob Roberts-Baca, and Yousef AbuHashem. Supported by Nick Bambos.

The Quick Version

We stress-tested BitTorrent in a controlled virtual environment to see how it handles network chaos. Three big surprises emerged:

Volatile networks hurt more than consistently bad ones. Switching between good and bad network states caused worse performance than staying in a bad state—BitTorrent needs stability to build effective peer relationships.

There's a cliff, not a slope. BitTorrent handles up to ~10% packet loss gracefully, then falls apart rapidly. Between 10-15% loss, performance doesn't degrade—it collapses.

More peers ≠ better performance (at least with one seeder). Contrary to P2P intuition, adding more downloaders just created more competition for the single source, with throughput dropping 71% as swarm size increased.

Key Takeaway

Distributed systems don't degrade gracefully—they have sharp breaking points. BitTorrent handles 10% packet loss fine, then collapses at 15%. Network volatility at specific frequencies causes worse performance than constant chaos. More peers don't always help if there aren't enough sources.

Introduction

Distributed systems depend on networks that don't always cooperate with one another. Whether it's peer-to-peer file sharing, blockchain propagation, or distributed storage, these systems must handle packet loss, fluctuating environmental conditions, and competition for limited bandwidth – whether explicitly or implicitly.

We used BitTorrent as a lens to study these dynamics—it's a well-understood protocol with clear mechanisms for peer selection and failure recovery, making it an ideal substrate for understanding how distributed communication breaks down under stress.

Figure 1: Client-server architecture (left) vs. BitTorrent's peer-to-peer model (right). In P2P systems, peers share pieces with each other, distributing load across the network rather than bottlenecking on a single server.

BitTorrent is elegant in theory: instead of everyone hammering a single server, peers share pieces of files with each other. The load distributes across the network, and everyone benefits from collective bandwidth. However, network reliability matters. Mobile connections drop packets. Congested infrastructure creates bottlenecks. Geographic distribution introduces latency spikes.

The principles we uncover—adaptive peer selection, timeout management, distributed coordination—apply broadly to CDNs, blockchain networks, and distributed storage systems like IPFS.

Methods

We built a complete BitTorrent testing environment using Mininet, a network emulator that creates virtual networks with precisely controlled conditions on a single machine. Unlike distributed testbeds like the AWS setup in our PropShare replication, Mininet runs all peers locally as virtual processes, allowing us to dial in exact packet loss rates, switch network conditions on command, and eliminate the random variables that make real-world testing messy.

Our Mininet setup ran up to 30 virtual peers sharing 4MB files, with one seeder (the peer with the complete file) and multiple leechers (peers downloading). We ran three experiments:

Experiment 1: Network Mood Swings

We made the network flip between "good" (1% packet loss) and "bad" (25% packet loss) states using a Markov chain model. The key variable: how likely is the network to switch states? We tested transition probabilities from 0.1 (stable) to 0.9 (chaotic).

Figure 2: Markov chain model for network volatility. The transition probability (p) controls how often the network switches between "good" (1% loss) and "bad" (25% loss) states. Higher p values create more unpredictable conditions.

Experiment 2: Steady Packet Loss

Simple and direct—what happens as we uniformly increase packet loss from 0% to 40%?

Experiment 3: Swarm Size

Under ideal network conditions, does adding more peers help or hurt individual download speeds?

Results

Network Volatility: Stability Beats Chaos

Figure 3: Download throughput as a function of network transition probability. The minimum occurs at p=0.8, not p=0.5 as expected—moderate-to-high switching rates are worse than maximum randomness because the protocol never stabilizes.

The results were counterintuitive. Maximum unpredictability (p=0.5, where either transition is equally likely) might be expected to cause the worst performance. However, the throughput minimum occurred at p=0.8—moderate-to-high switching rates were worse than maximum randomness.

BitTorrent's peer selection algorithm (read more about unchoking and choking here) runs on 10-30 second cycles. It needs time to evaluate which peers are worth exchanging data with. When network conditions change every few seconds, the protocol never gets a chance to establish productive relationships. This points to a larger point that Nick Bambos, our professor, so rightfully made: fluctuations really are the root-cause for all this complexity in networks and distributed systems. This intuitively makes sense; if there were no fluctuations in the networks, there would be a very clear policy one could follow for their networking/distributed-systems environment. Even a few fluctuations are 'fine', assuming some mediocre ability to adapt to changing external environments. But environments rapidly changing all the time? That doesn't really leave any breathing room to fall into steady states...

Performance by Transition Probability:

Packet Loss: The 10% Cliff

Figure 4: The 10% packet loss cliff. Performance degrades linearly from 0-10% loss, then collapses catastrophically beyond 15%. The 2-second request timeout becomes a bottleneck when too many requests time out simultaneously.

BitTorrent handled packet loss gracefully up to a point. From 0-10% loss, throughput declined linearly and manageably (27 KB/s down to 15 KB/s). Beyond 10%, performance collapsed rapidly. At 15% packet loss, throughput crashed to 2.7 KB/s. By 30-40%, the protocol effectively stopped working.

This breakdown likely occurs because BitTorrent's 2-second request timeout becomes a bottleneck. At low loss rates, occasional retries are fine. But once too many requests start timing out, the protocol spends more time waiting than transferring. The safety mechanism becomes the bottleneck.

Swarm Size: The Single-Seeder Trap

Figure 5: Monotonic decline in individual throughput as swarm size increases (single-seeder scenario). With only one source, more leechers means more competition for limited bandwidth—peer-to-peer magic requires multiple seeders to work.

The results contradicted the expected pattern. An inverted-U curve might be expected—small swarms suffering from limited piece diversity, large swarms suffering from coordination overhead, with a sweet spot in the middle.

However, the data showed monotonic decline. One leecher got 107 KB/s. Five leechers got 80 KB/s. By 25+ leechers, everyone was stuck at ~30 KB/s.

The problem is fundamental: peers can only share pieces they've already downloaded. With a single seeder, every piece must originate from one source. More leechers just means more competition for that limited bandwidth. Peer-to-peer systems require multiple seeders to function effectively.

Insights

Distributed systems have sharp thresholds rather than gradual degradation. BitTorrent handles 10% packet loss gracefully, then collapses beyond 15%. Network volatility at moderate-to-high switching rates (p=0.8) hurts more than constant chaos because the protocol needs stability to build peer relationships.

The single-seeder trap shows that P2P systems only work when there's actual peer-to-peer exchange. More participants don't automatically improve performance—they need resources to share. This applies beyond BitTorrent to any distributed system relying on collaborative resource sharing.

Protocol design matters. Fixed timeouts and unchoking intervals work well under normal conditions but become pathological under stress. Adaptive mechanisms—timeouts that adjust to packet loss, unchoking that responds to volatility—could make distributed protocols more resilient.

Conclusion

BitTorrent's breaking points are predictable and exploitable. The 10% packet loss cliff, the single-seeder trap, and the volatility problem aren't bugs—they're inherent limitations of the protocol's design choices. Understanding where systems break helps operators set realistic boundaries and guides protocol designers toward more adaptive mechanisms.

The principles here extend beyond file sharing to blockchain propagation, distributed storage, and CDN architectures. Any system coordinating resources across unreliable networks faces similar tradeoffs between simplicity and adaptability.

Acknowledgments

This project was a collaborative exploration of how distributed systems fail under stress. Thanks to Jacob Roberts-Baca and Yousef AbuHashem for the countless hours debugging Mininet configurations, analyzing performance data, and uncovering these surprising thresholds together. Special gratitude to Professor Nick Bambos for guidance on experimental design and performance analysis methodology—his insights on finding and characterizing system breaking points were invaluable.

The team after presenting our findings. From left to right: Jacob Roberts-Baca, Yousef AbuHashem, Shounak Ray, and Professor Nick Bambos.