Review1Add my vote for this tag create new tag
view all tags

Paper Review: Is There WiFi Yet? How Aggressive Probe Requests Deteriorate Energy and Throughput at IMC 2015

Paper Info

X. Hu, L. Song, D. Van Bruggen, and A. Striegel, "Is There WiFi Yet? How Aggressive Probe Requests Deteriorate Energy and Throughput," in Proc. of IMC 2015, Tokyo, Japan, Oct. 2015. PDF Review Slides


These reviews are provided verbatim for those also working in the area. The paper itself was shepherded and appeared as a short paper at IMC.

                           IMC 15 Review #16A
                    Updated 19 Jun 2015 7:28:07pm EDT
  Paper #16: Is There WiFi Yet? How Aggressive Probe Requests Deteriorate
             Energy and Throughput

                     Overall merit: 2. Weak reject
                Reviewer expertise: 3. Knowledgeable

                        ===== Paper summary =====

This paper studies the volume of WiFi probes using multiple datasets from football games and classes, seeking to demonstrate that aggressive probes in these environments are unnecessary, e.g. no WiFi AP is nearby.  The authors then built a set of small local experiments (in the lab) to examine the impact of (aggressive) probe on energy and network performance.

                     ===== Comments for author =====

The paper abstract/intro are misleading. The dynamics of WiFi probes were studied using a large dataset but the impact on energy and throughout was examined using a small in-lab setup. yet the authors presented the paper as they studied all these issues in both large and small datasets.

The dynamics of the probe traffic are interesting but the current paper is unable to draw a line between the probe traffic volume to the network performance. 

The fact that WiFi probes cost energy/overhead is well known and there have been extensive works on mitigating such issue, e.g.: blue-Fi uses cellular and bluetooth signature to keep WiFi off while performing AP discovery. 
- Blue-Fi: enhancing Wi-Fi performance using bluetooth signals, in MobiSys 09, 
- Footprint: cellular assisted Wi-Fi AP discovery on mobile phones for energy saving, WINTech’09  
Given these existing works, any new measurement study should provide new insights and use accurate power breakdown (using various energy models). This work did not do so.

                           IMC 15 Review #16B
                    Updated 23 Jul 2015 6:21:10pm EDT
  Paper #16: Is There WiFi Yet? How Aggressive Probe Requests Deteriorate
             Energy and Throughput

                     Overall merit: 4. Accept
                Reviewer expertise: 2. Some familiarity

                        ===== Paper summary =====

The paper describes some measurements of 802.11 probe
request behavior in crowded domains (a stadium and a large
lecture hall), both with (lecture hall) and without
(stadium) a public (enough) infrastructure.  The authors
characterize the inter-probe request time before iOS8, where
mac addresses were not randomized, and observe the
performance implications (interference) of devices that scan
frequently and at low bitrates.

The experiments (tracing at a few events; a basement
power-consumption study) seem preliminary in their scale and
rigor, and not that well explained, but likely adequate for
a short paper.

                     ===== Comments for author =====

It is possible that reviewers with greater expertise will
recognize aspects of this paper that are not novel.  I
believe it is known that scanning leads to power
consumption.  However, the varied behaviors shown in table
2, as well as the detail about iOS8,
screen-activity-dependent scanning, and a throughput
measurement to verify that probe requests have an effect,
are new to me.

"The reality though is that the Probe Request consumes an
impact of much more than the 800 microseconds of air time."
Please calculate or measure the expected actual contention
time.  The rest of the section seemed weak; as if the 800
microseconds are somehow different than 800 used for a
different wireless purpose - of course there's
contention.. What's ANDSF?  Just a sentence or two would be

Figure 1 (the probe requests over time graph) doesn't
obviously teach much.  There is perhaps a relationship where
the graph shows how many users are turning on the screen of
their phones; perhaps annotating the graphs with time-outs or 
other events would be helpful.  

Figure 2's x-axis should have a label at 0.  Most data are
between 0 and 10, so the 0 label is important.

Figure 3's superimposed histogram is not useful; actually
raising questions since the bar must be offset.

Section 3.2 - It's not clear to me why the cellular network
usability would alter scanning behavior of the client for
802.11.  Is there an experiment that could be run to validate
this conjecture about increased scanning rate on game day?

Is there something more to be said about the different
probing approaches of Android vs iOS?  Is it better to cram
all the probing into a short period of disruption or spread them 

Section 4.2 - it's a bit hard to believe that a basement is
sufficient isolation for a controlled experiment; it would
help to explain that this isolation is sufficient for this

Table 3 - why use thousands of micro amp hours instead of
milli amp hours?

Section 5 was quite nicely written.

Nit: I didn't notice where UE was defined.  It seems used
first in 3.1.  Probe Request is expanded twice (PR) (third
paragraph, also first bullet)

                           IMC 15 Review #16C
                    Updated 26 Jun 2015 6:55:37am EDT
  Paper #16: Is There WiFi Yet? How Aggressive Probe Requests Deteriorate
             Energy and Throughput

                     Overall merit: 4. Accept
                Reviewer expertise: 4. Expert

                        ===== Paper summary =====

This paper is looking at the "cost" of discovering WiFi networks. The authors have collected data from super-dense events, like sports events, from a large classrooms, and have also performed controlled experiments in the lab. They clearly demonstrate that the density of ProbeRequests is very significant in environments with a high density of WiFi clients, and that leads to significant energy and throughput cost.

                     ===== Comments for author =====

I think that this is a very nice short paper for IMC. It starts with a very specific problem and tries to understand its extent. I really like the fact that the authors have collected measurements from different super-dense events, and that they have also collected measurements from a classroom, thus quantifying the impact of the existence of a WiFi infrastructure that could mute such requests. The frequency of Probe messages is indeed disconcerting when it comes to high densities of clients and the reality is that there is not an obvious fix to the problem reported.

The main weakness of the paper is that the authors have never really studied the way that discovery happens in the 3 types of devices they use. Yes, they show that when the WiFi discovery app is ON then most clients will probe every 10 seconds. However, they have not studied whether these devices implement particular probing strategies. The assumption made in the paper is that the devices will send a ProbeRequest in every channel in sequence. In reality that may not be the case, and there has already been research on how best to hop across channels to optimise discovery time. I would have liked to see the authors testing the actual mechanism implemented in their devices. What would this mean?

You get each one of the devices and you collect data on how they scan the frequency bands (in what channel order), how they use PNL, and how they switch from 2.4GHz to 5GHz. Let that process last for a few minutes to understand whether they exponentially backoff when they do not find an available AP. The measurements presented show that when the WiFi discovery app is OFF the intervals may go up to 240 seconds. I would have liked to see an exploration of that sort. That would allow you to understand things much better. Granted that each operating system will do something differently but at least this way you can better understand how much current implementations cover the entire design space of options. 

I believe a study of this sort should form part of the camera ready version of this paper if the paper ends up being accepted.

In such a case, I think you should also tone down a little your claims on "costs". Most of your costs right now are worst case. You assume that there is going to be continuous scanning every 10 seconds. But, in real experiments you actually see the interval between continuous scans to be much greater when the device is not in active discovery mode, but does so in the background. Offering a range of energy cost and throughput cost would provide a more balanced viewpoint.

In terms of solutions, one easy fix to the problem reported is to use the geofencing API. If the previous scan has not led to a list of APs, and the device has not significantly  moved, then the device would not rescan. That could work in areas where there are no WiFi APs, and could also possibly improve a little bit the case when the scanning is opportunistic aiming to identify better quality APs.

But all in all, I find that the paper is indeed making a useful contribution. In particular, since ultra-dense environments have not been extensively studied in the past. I would be happy to see this paper presented at IMC.

                           IMC 15 Review #16D
                    Updated 13 Jul 2015 1:09:52pm EDT
  Paper #16: Is There WiFi Yet? How Aggressive Probe Requests Deteriorate
             Energy and Throughput

                     Overall merit: 2. Weak reject
                Reviewer expertise: 2. Some familiarity

                        ===== Paper summary =====

This paper conducts a study of WiFi scans in two environments, a stadium with no WiFi, and a large classroom with WiFi.  It attempts to make an argument that these scans both cost significant energy and have an important impact on WiFi performance, but these arguments don't convince me.  

Most importantly, they don't offer or evaluate any solutions.  Scans happen every 10 seconds or so.  If that's so bad, what should phones do, exponential backoff?  That's a simple policy that could be easily evaluated, with clear advantages and disadvantages.

                     ===== Comments for author =====

What is the campus DAS?

Table 1: Can you include median or stdev numbers for the last two rows? I'm curious about the underlying distribution.

Section 3:

Figure 1 doesn't really tell me anything interesting.  The CDFs in Figure 2 are better.  Something very useful would simply be looking at the scan behavior as a function of operating system type and version.  Maybe some get it right an others don't?  Table 2 scratches that surface but it is not enough.

Section 4.1: Sorry, I am not buying it.  While 7% wastage is not great, arguing that 80K phones is not compelling.

Section 4.2:

Why exclude the 2.4 GHz results if they are worse?  Use them instead!

Can you scale up incrementally to get a sense how that curve looks.
Exponential vs. logarithmic  is an important distinction.

While there's not a lot of "ultra-dense" WiFi papers, you did miss these:

Gupta, Arpit, Jeongki Min, and Injong Rhee. "WiFox: Scaling WiFi performance for large audience environments." Proceedings of the 8th international conference on Emerging networking experiments and technologies. ACM, 2012.

Yiakoumis, Yiannis, et al. "BeHop: a testbed for dense WiFi networks." Proceedings of the 9th ACM international workshop on Wireless network testbeds, experimental evaluation and characterization. ACM, 2014.

Manweiler, Justin, and Romit Roy Choudhury. "Avoiding the rush hours: WiFi energy management via traffic isolation." Proceedings of the 9th international conference on Mobile systems, applications, and services. ACM, 2011.

Minor comments:

- "IETF 2006" --> "i.e., IETF 2006"
- "due consistent rain" --> "due to consistent rain"

                           IMC 15 Review #16E
                    Updated 12 Jul 2015 7:47:56am EDT
  Paper #16: Is There WiFi Yet? How Aggressive Probe Requests Deteriorate
             Energy and Throughput

                     Overall merit: 2. Weak reject
                Reviewer expertise: 2. Some familiarity

                        ===== Paper summary =====

The paper analyzes the dynamics of 802.11 PR and its impact on energy and throughput. As providers and devices push users to WiFi for offloading and, to improve joining time, requiere more aggressive probings, the spotting availability of WiFi means that much of those probings and their associated costs go to waste. The paper study this point using data collected in three different settings, a large stadium, a classroom and a lab.

                     ===== Comments for author =====

This is an interesting read on the impact of PR rates and there is, perhaps, something to it based on anecdotal evidence of batteries running low in places with poor/no WiFi. Still, there is not enough here for an IMC paper - the scenarios are limited and the conclusions are seemingly weak. It’s hard to get excited about a hypothetical 7% battery charge loss throughout a 3hrs game in a fill stadium or an 8% loss in throughput with 5 other devices (does it grow with numbers of devices, what is the shape, is it only due to PR rates?). The presentation can also use a bit of work with many of the graphs introduced without much discussion and several terms used before being defined.

Why the high rate of PR in large spaces? What drives it? What would be the “right” way of doing it? Is there one? How would you/could you implement it? 
In the stadium I wouldn’t say the space is ‘bereft’ of wifi; how many devices are enabled as hot spots? Could that explain the higher rate? More devices, higher probability of finding more mobile hot spots, higher PR rates?
What is a “legitimate WiFI deployment”? Would this hotspots not be?
Figure 1 results are presented making little to no inference from them; what do we care? what do they mean in the space? 
Besides, why not just presenting rate rather than forcing the reader to do the mental calculation?

What does it mean that ‘the constant number of Probe Requests is implied by the stable population in the bowl, which in turn can serve as an indicator of PR prevalence when individuals are seated”? I honestly can’t interpret this….

What is the significance of non-wildcard, wildcard SSID? why do we care? What is PNL? (this is only defined in page 5!)
What’s the goal of the low-pass filter? 

The comment about iOS8 problems (changing MAC) is not explained the fourth page but it is used in every page before. You need to explain this issue earlier.

I am having a hard time with the power consumption back-of-the-envelope calculation. It feels as if you are going really out of our way to make it significant - consumption is measured for an active scan and from there, you go to imagined scenarios for 10’, a game, and then the whole stadium! 

How do you determine that the only reason for the 8% drop in throughput is ‘useless’ PR? 


- large conference venues - remove ‘venues’, add ’s’ to conference
- surmise that such - remove ‘that’
- due “to” consistent rain and … (missing ‘to’)
- class room -> classroom
- to explore the data further we divide into the data as observed -> we divided base don the general location of the observations ?
- may in part be driving the cellular network on campus being overwhelmed … rephrase, it doesn’t make sense 
- “described as follows” would mean you are describing how you describe it, probably not what you mean
Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | More topic actions
Topic revision: r1 - 2015-10-29 - AaronStriegel
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback