University of Notre Dame NetScale Laboratory

Navigation

Webs Webs

Main » Projects » ProjectTransBWC » TransScaleBoxPos » TransScaleBoxPosReviewResponse

ScaleBox - Detailed Review Response

Preface

The review response is provided as a response to the reviews received from HotNets VI for the paper concerning ScaleBox and its implications for scalability and centralization. You may find the original reviews along with the paper itself here on the wiki as well.

Reviewers are welcome to post anonymously to my blog to encourage further dialog on the topic with your responses being included below.

Response Status: Currently mid-way through the second review for the responses.

Review / Responses

HotNets VI Review #48A

Reviewer:	First, I found the paper so hard to read that I often really couldn't tell just how various mechanisms work. From what I could tell, however, it appears that quite a bit of the scalability gain is simply the use of compression to reduce content repeated to multiple recipients (along with some extra hacks whose correctness/utility is not established). This has problems, as I frame below.
Response	The lack of understanding of the mechanisms was a failure on our part as presenters. Our assumption regarding that reviewers would follow the citations for our previous work on stealth multicast and packet caching was incorrect. A figure explaining the techniques was likely warranted or perhaps expanding the ScaleBox figure to a full page width with break out panels would have been in order trading parts of the discussion on TCP or the time window gains for coalescing. The phrasing of "extra hacks" is perplexing as unless the reviewer views Akamai as a hack, all mechanisms are fully in line with underlying protocols. Moreover, very little changes in the network beyond actually using multicast if available with zero changes occurring at the client. As we emphasized continually in the paper, it does not require a clean slate to do this and our previous work with EEOD showed at least a 10x gain for a dynamic website.
Reviewer:	In addition, the paper doesn't offer any empirical evidence regarding the degree to which compression opportunities will arise (other than "Recent examinations of the university tap" without any details).
Response	The reviewer is correct in that we do not offer hard empirical evidence directly in the paper. However, we would like to note to the reviewer the citation for our EEOD paper showing improvements of 10x and beyond for websites such as CNN, Amazon, and Slashdot. Richer websites such as our most recent university website at Notre Dame will benefit even more. Moreover, we also point to the work by Cheshire showing that similarity (i.e. redundancy) tends to emerge the most when the traffic is the heaviest. As noted in the paper, the technique offers the most benefit when it is needed the most. We currently have large scale analyses underway but those works are computing at the moment with an estimated completion time for our paper at SIGCOMM.
Reviewer:	"private user goodness" and "what immerses the user" in the first paragraph are both confusing phrases.
Response	Private user goodness was a bit casual but in short, the good stuff that the user selects or customizes is what drives Web 2.0. Dynamics are simply a fact of life and drive the immersion, i.e. the perception of ubitiquous computing.
Reviewer:	Please provide a citation for YouTube? finding centralization a necessity.
Response	The phrasing was perhaps a poor choice on our part. In short, we thought that one could naturally follow that the sheer breadth of scale and popularity negates YouTube? from embracing the classic CDN type of operations, i.e. distribute my content across your vast network. While there was a nice article in Data Center Knowledge (September 12, 2007), it came out a bit late and in general we try to avoid citing purely on-line articles. The notable quote in the article - "Level 3 Communications will provide big-pipe connectivity to help the video portal YouTube? keep up with its rapid growth, the two companies said today. Under the terms of the agreement, Level 3 will speed YouTube? 's video delivery with high speed Internet access through multiple 10 gigabits per second (10GigE) ports across a nationwide fiber network connecting YouTube? 's data centers."
Reviewer:	For on-line games, centralization is driven not only by security/quality controls, but (arguably much more so) by interactivity, which can't be achieved in a loosely coupled distributed manner.
Response	This is simply incorrect. Ignoring the obvious cases of MMORPGs ala World of Warcraft, EVE Online, etc. where security / quality control is paramount to the user experience, there is a long and storied history of users electing to cheat whenever given the option. The original Counterstrike mod to the game Half-Life was a perfect example with many notable speed, targeting, and general cheats incorporated by players when logic is deferred to the client-side host. PunkBuster? the anti-cheating tool in large evolved to detect client-side cheating and Intel has been exploring efforts on how to control client-side cheating. There are also notable games that allow for fully distributed computing in the RTS arena (Supreme Commander) that validate games at a central server to detect cheating for ranked ladder matches. Plain and simple, it is quality and security now that dominates with reasonable speeds for broadband networking for most on-line players (receive speed, not sending speed).
Reviewer	"scurrilous" is a pretty strong word, and likely not really what the authors wish to convey about their postulation.
Response	Given the reaction of the reviewers, it seems quite apt
Reviewer	In 2, the sentence "For the purposes of simplicity" appears twice.
Response	Grammar / proofing mistake. Yes we wanted to emphasize it but not quote like that
Reviewer	Caching on a per-packet basis seems to not be a great granularity, since the same content can easily be packetized along different boundaries (e.g., due to slightly different length HTTP headers).
Response	The authors would like to note to the reviewer the following phrase that appeared in the packet caching discussion "Ideally, data would be separated on a packet-wise basis via techniques such as the TCP Explicit End of Data (EEOD) \cite{EEOD:HotWeb06} (minor server application modification/kernel modification) to enable the use of whole packet caching." The entirety of the EEOD work (cited in the paper that appeared in HotWeb? 2006) focused on how to deliver such separation in a simple manner, specifically geared towards HTTP serving addressing that exact issue.
Reviewer:	In the discussion of Stealth Multicast, which nodes are the multicast receivers? How do they decide to join the group?
Response	While these details are tied to the specific transport mechanism utilized in stealth multicast (broadcast, PIM, ALM), the reference our work on PALM - Passive Application Layer Multicast and the phrasing "In particular, our work in \cite{CompNet07:PALM} proposes an ALM-centric version of stealth multicast (PALM - Passive Application Layer Multicast) whereby a node at the client-side ISP would serve as a ALM Helper Device (AHD) to send/replicate data and convert PALM data packets to the original data packets." roughly describes one place for conversion. While conversion could take place in the immediate AS following the server at the egress point, an approach such as PALM make much more sense.
Reviewer:	More generally, the scheme appears to require a zillion bilateral configurations between ISPs and servers. How can that scale management-wise? How can an ISP possibly arrange deals with all of the servers with which its customers might engage?
Response	We note that we encourage fluid requests for resources rather than explicit bilateral configurations. It is in the economic interest of the downstream ISP to encourage this to reduce their incoming bandwidth and hence the relative disparity of data served versus data received. A dynamic discovery protocol which is already necessary would discover such devices on a periodic basis. Moreover, we note that one does not have to cover all customers but only the heaviest of hitters as well requiring a few less than a zillion bilateral configurations.
Reviewer:	Why in figure 3 were the 3WHS, HTTP request, and connection closure ignored?
Response	The removal of 3WHS (3-way handshake) and the HTTP request was a quirk of document preparation. Connection closure has little to no bearing on the perceived rendering time and hence is fairly ignored. While the addition of those items would increase the FCT, we note that we picked fairly low volume web sites for an example along with a relatively conservative look ahead window.
Reviewer	TCP Vegas is not a "more aggressive algorithm".
Response	Our phrasing was less than precise as the implication was that with an accelerated / proxied ACK under TCP Vegas, the algorithm would react aggressively to sudden changes in RTT.
Reviewer	"Rather than tying replicated contact to explicit agreements" - businesses generally want explicit agreements in matters concerning their resource allocations.
Response	In today's current environment, we would agree that this is the perception. However, this is also in large part to how the CDNs themselves have evolved in that the CDN covers a massive swath or the entirety of the Internet and one negotiates an explicit agreement with the CDN (Akamai, Savvis, etc.). When content has limited breadth, this is entirely realistic. In contrast, the trends for content is not towards single source but rather increases and breadth and richness. Take the example we continually highlight in the paper, YouTube. YouTube is a paltry 320x200 movie versus average video camera capacities far exceeding that. Tie in HDTVs and real bandwidth to the home (both uplink and downlink) and while storage is cheap, co-locating many sites locally will only get harder. In contrast, we argue that the bulky CDN model is the wrong approach and that resource allocations for localized resources should be fluid, i.e. a content provider can dynamically negotiate as necessary. In some sense, it follows the trends in FIND and GENI with dynamic light paths, resources should be more easily configured, perhaps not instantly but should not be static. Moreover, as we argued above, it is in the economic interest of the local ISP to offer such a service. Allowing for dynamics allows for less downstream bandwidth during bursts and more highly congested periods.

HotNets VI Review #48B

Reviewer	The techniques used is the ScaleBox are not terribly novel, and the details of how they make things work are not terribly clear.
Response	While we would concur that the presentation could use improving, the reviewer incorrectly assesses the novelty of the paper. As will be noted below, the tail synchronization approach is unique in several aspects versus previous VoD? approaches. Most notably, the environment of tremendous breadth (i.e. a YouTube) and expectations of instant streaming make the multi-streaming aspects of a VoD? approach.
Reviewer	There is also something about the writing style of this paper that really rubbed me the wrong way. For example: "ScaleBox offers something of an anomaly in the network world, scalability and performance despite centralized serving akin to the colloquial phrase of having one's cake and eating it too."
Response	The casual phrasing in the paper was an incorrect assumption of rapport with the reviewer and best left to a presentation rather than in the paper. We incorrectly assumed a more casual setting with versing similar to papers from previous HotNets.
Review	You need explain how "tail streaming" is either different than or the same as stream merging, a well-known technique developed for streaming media. See "Minimizing Bandwidth Requirements for On-Demand Data Delivery" by Derek Eager, Mary Vernon, and John Zahorjan in IEEE Transactions on Knowlege and Data Engineering, 2001.
Response	Tail synchronized media focuses on creating the maximum amount of isolation between the data being attempted to be efficiently streamed and the forward buffers that are responsible for avoiding buffer emptying. As the typical home environment with all of its NATs, asymmetry, and most recent bandwidth/delay shaping show, setting up a group-wise distribution scheme is likely to be problematic. Moreover, outside of a nice symmetric environment where many protocols are developed, the capacity for downward streaming pales and simply cannot satisfy the forward playback requirements. For long-lived flows or longer media, the work noting in the Eager, Vernon, and Zahorjan work (stream merging via HMSM) as well as the numerous VoD? works in the literature are sufficient as playback timeliness is not a requirement, i.e. one simply needs to get the content not satisfy playback constraints. Furthermore, while merging in an IP multicast case (as would likely be done with the above paper) is fairly straightforward, merging in an ALM, SGM, or end host / end ISP setup is not as straightforward. It is our premise that setting up the tree once is simpler from a management perspective as well.

HotNets VI Review #48C

Reviewer	While the core ideas appear interesting, the paper does a poor job of defining the architecture it is proposing clearly, arguing and quantifying what the benefits are, and explaining what incentives there are for ISPs and content providers to deploy this architecture.
Response	As noted above, we presumed that the references for the base of the architecture would be read as well to address new material rather than re-hashing previously published papers. The observation of the lack of explanation regarding benefits is curious as a key aspect of the paper is how ScaleBox gets the economic incentive right, i.e. those parties that are most likely to benefit are the only ones needing to modify their network.
Reviewer	The paper is also very poorly written which makes it hard to understand what is being proposed.
Response	This was clearly a mistake on our part on presentation.
Reviewer	The whole motivation of offering centralized content distribution (in the introduction) is confusing and is irrelevant to the archtiecture being proposed.
Response	We disagree as the counterargument to ScaleBox is that one should simply use a CDN, i.e. they already cover what is needed. We posit that there exists significant more potential for dynamism in the network than is currently being employed, in large part due to the CDN nature of the network.
Reviewer	The largest problem I had with this paper was the writing often obscured the ideas being presented. While the ideas presented might be be good the presentation did not convince me of that.
Response	Again, our mistake as the authors to not clarify our presentation better.
Reviewer	I would try to more systematically define the parts of you architecture and the gains that are obtained from different components. Also I would like to see a more concrete comparison between the three approaches talked about in the paper (Centralized, CDN, ScaleBox) and what the tradeoffs between each are under different situations.
Response	These were stated more concretely in the references for stealth multicast and packet caching. We should have likely used graphs from those works to better reflect the benefit rather than including the newer material.
Reviewer	At times I felt like you were making more philosophical arguments rather than technical ones. While this may be fitting for a workshop such as HotNets? , I feel like your bold statements need to be better grounded in a technical basis.
Response	Correct. We misinterpreted our rapport and ability to argue philosophically was based on the reviewers thoroughly following the core reference works (stealth multicast, caching). A technical basis was available but elected not to be included.
Reviewer	I would go through your paper and make sure that all terms you use are clearly defined. You mention the term centralized distribution in the third paragraph of the introduction. I have no idea what you mean by this and this left me confused for a while. I decided that you meant centralized at one site, but distributed over multiple machines. Another example is the introduction of the term "mice" packets, maybe this is a common term I'm not familiar with but I was confuse by this as well.
Response	The terms are both fairly common networking terms from traffic engineering but it was our mistake to assume complete knowledge of the reader.
Reviewer	In the body of the paper I felt like you were bringing up ideas and then immediately moving on to the details of the implementation. If you could explain each of the components at a high level and then have subsection that discuss them in more depth this might make it more clear to me what exactly you are doing. For example the text under heading 2 could have been expanded and some of the other details later in the paper omitted. I would suggest having subsections for "Goals", "Components", "Packet Caching", and "Stealth Multicast". I would spend much more time on these sections as they are the basis of your idea. You spend a huge amount of space discussing TCP fairness in 2.1, which I felt while important, was tangential to your main point.
Response	Our goal with the paper was to discuss philosophical aspects, not to simply rehash our existing work. While we certainly could have done so and likely to the great benefit of reviewers, the work has already been published in large part.
Review	Some of the 2.4 points should be discussed earlier. For example, I was asking the question "Isn't this merely a camouflaged CDN?" very early in the paper. If you could anticipate this earlier and head it off. Perhaps in the first part of 2 or in the introduction. Again as I said earlier, perhaps a section comparing the architecture of CDNs, Centralized and ScaleBox could be added which would make this point crystal clear.
Response	The problem with mentioning these points is that the context of the questions would have been largely lost. In some sense, the context was lost in the first page but the questions were only valid after reading most of the paper rather than placing them early on.

r3 - 23 Oct 2007 - 19:01:12 - AaronStriegel

Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback

Syndicate this site RSS ATOM