In this example, we look at how we can send multiple HTTP requests and
process them concurrently. The basic idea is this. We specify a
collection of URIs to download when establishing the HTTP requests, but
don't send any of the requests until they are all constructed. Then, we
send the requests and harvest the results.
When the last one finishes, we return control to the caller.
This is different from processing the requests in the background. In
that case, having dispatched the requests, control would be returned
to the caller and R would be able to do other things. Notification of
pending content from the request would be done via the event
loop. This is feasible (at least on Unix), but different
from the example we are describing here.
The implementation requires using the multi interface for libcurl. We
create a multi handle and then we create a regular curl handle for
each individual request, i.e. for each URI to be fetched. We add each
of these regular/easy curl handles to the multi handler and then call
curlMultiPerform()
until it terminates. Terminating
means either an error or that each of the requests has completed.
getURIs =
function(uris, ..., multiHandle = getCurlMultiHandle(), .perform = TRUE)
{
content = list()
curls = list()
for(i in uris) {
curl = getCurlHandle()
content[[i]] = basicTextGatherer()
opts = curlOptions(URL = i, writefunction = content[[i]]$update, ...)
curlSetOpt(.opts = opts, curl = curl)
multiHandle = push(multiHandle, curl)
}
if(.perform) {
perform(multiHandle)
lapply(content, function(x) x$value())
} else {
return(list(multiHandle = multiHandle, content = content))
}
}
To test this, we can fetch pages from the Omegahat Web site:
getURIs(c("http://www.omegahat.net/index.html", "http://www.omegahat.net/RecentActivities.html"))
To see what effect we have on timing, let's run the asynchronous
version and the serial version and compute their times.
uris = c("http://www.omegahat.net/index.html", "http://www.omegahat.net/RecentActivities.html")
atime = system.time(z <- getURIs(uris))
stime = system.time(zz <- lapply(uris, getURL))
How does this vary with larger numbers of URIs?
uris = c("http://www.omegahat.net/index.html",
"http://www.omegahat.net/RecentActivities.html",
"http://www.omegahat.net/RCurl/index.html",
"http://www.omegahat.net/RCurl/philosophy.xml",
"http://www.omegahat.net/RCurl/philosophy.html")
atime = system.time(z <- getURIs(uris))
stime = system.time(zz <- lapply(uris, getURL))
The key thing to note is that the user and system CPU times are much
lower for the serialized downloads, but the overall elapsed time is
significantly higher. This is quite reasonable when one thinks about
it. The asynchronous version spends time switching between the
downloads, checking if there is anything to be read on each. The
serialized version does not have to do this switching but can focus on
each single download. The switch is done in a call to
select. This can be arranged not to consume
CPU time, if I recall correctly, but it will do so if a timeout is
specified.
This tradeoff between elapsed time and CPU time is a convenient
one. The user can chose which is important for their particular situation.
Being statisticians, we should repeat the measurement process
and get data about the variability of the times taken in each
approach.
atimes = sapply(1:40, function(i) system.time(getURI(uris)))
stimes = sapply(1:40, function(i) system.time(getURI(uris, async = FALSE)))
times = data.frame(user = c(atimes[1,], stimes[1,]),
system = c(atimes[2,], stimes[2,]),
elapsed = c(atimes[3,], stimes[3,]),
style = factor(c(rep("Asynchronous", 40), rep("Serial", 40))))
by(times, times$style, summary)
times$style: Asynchronous
user system elapsed style
Min. :0.340 Min. :0.0300 Min. :0.430 Asynchronous:40
1st Qu.:1.665 1st Qu.:0.1275 1st Qu.:2.085 Serial : 0
Median :2.135 Median :0.1700 Median :2.865
Mean :2.319 Mean :0.1790 Mean :3.091
3rd Qu.:2.740 3rd Qu.:0.2025 3rd Qu.:3.683
Max. :5.780 Max. :0.5400 Max. :7.410
------------------------------------------------------------
times$style: Serial
user system elapsed style
Min. :0.0500 Min. :0.00000 Min. : 1.780 Asynchronous: 0
1st Qu.:0.0875 1st Qu.:0.00750 1st Qu.: 3.788 Serial :40
Median :0.1050 Median :0.01000 Median : 5.320
Mean :0.1168 Mean :0.01325 Mean : 5.973
3rd Qu.:0.1225 3rd Qu.:0.02000 3rd Qu.: 7.875
Max. :0.2500 Max. :0.04000 Max. :14.020
As usual, a picture makes things clearer. A
stripplot()
of elapsed time shows us the
different distributions for the serial and asynchronous transfer.
Clearly, the download times for the asynchronous
download are significantly shorter than for the serial
download. And the variability is also much larger.
Stripplots for user and system times show the
difference between the two modes of downloading also
and that the asynchronous method makes more use
of system resources.