FastReplica: Large File Distribution within CDNs

Authors: Ludmila Cherkasova, Jangwon Lee

Complete Citation

L.Cherkasova, J. Lee. FastReplica: Efficient Large File Distribution within Content Delivery Networks. Proceedings of the 4th USENIX Symposium on Internet Technologies and Systems, Seattle, Washington, March 26-28, 2003.

Abstract

In this work, we consider a large-scale distributed network of servers and a problem of content distribution across it. We propose a novel algorithm, called FastReplica, for an efficient and reliable replication of large files in the Internet environment. There are a few basic ideas exploited in FastReplica. In order to replicate a large file among n nodes (n is in the range of 10-30 nodes), the original file is partitioned into n subfiles of equal size and each subfile is transferred to a different node in the group. After that, each node propagates its subfile to the remaining nodes in the group. Thus instead of the typical replication of an entire file to n nodes by using n Internet paths, connecting the original node to the replication group, FastReplica exploits n×n Internet paths within the replication group where each path is used for transferring 1 nth of the file. We design a scalable and reliable FastReplica algorithm which can be used for replication of large files to a large group of nodes. The new method is simple and inexpensive. It does not require any changes or modifications to the existing Internet infrastructure, and at the same time, it significantly reduces the file replication time as we demonstrate through experiments on a prototype implementation of FastReplica in a wide-area testbed.

Annotations

While the over-large abstract gives most of the details, a few things are worth pointing out. First, the system uses an n-to-n distribution scheme. Essentially they perform a type of software multicast. We are interested in this work as the Tail-Sync project will be software-multicasting the back end of the files it is distributing. Also, FastReplica contains backups in case a node fails, something we will need to consider in Tail-Sync.

-- DavidMoore - 05 Sep 2007