Metrics at a glance

As we overview in [SIGCOMM-QoE-16], there are basically four classes of WebQoE metrics:

Family Class Examples
Objective Time instant TTFB, TTFP, DOM, PLT, ATF, AATF, etc.
Objective Time integral SpeedIndex, ByteIndex/ObjectIndex, PSSI, etc.
Objective Compound scores Yslow, PageSpeed Insights, dynaTrace
Subjective User feedback MOS, user-PLT, acceptability, etc.

Metric of the subjective class require human feedback, whereas objective metrics can be directly computed by the browser software. Only in very few cases, some objective metrics can be directly computed by network devices. For instance, user impatience [GLOBECOM-03], or the ByteIndex metrics defined in [SIGCOMM-QoE-16], can be also measured from encrypted traffic [SIGCOMM-19].

In our work, we focus on defining objective metrics of the time instant and time integral classes, as well as gathering user feedback, to further develop models of user experience. Below, we explain the main concept in a nutshell, but please go through the publications and code sections to know more.

Time instant metrics

  • W3C Navigation timing metrics
Metrics in the time-instant class are those defined in the World Wide Web Consortium (W3C) Navigation timing) recommendation, that basically tracks a number of specific events reported in the classic (non-normative) illustration reproduced below. These metrics being part of a standard, and having a long history, they are traditionally available across all Web browsers.
  • Approximated Above the Fold (AATF)

However, whereas the onLoad is very often used to assess Web performance, it is quite well know that onLoad is simplistic and does not faithfully represent the quality of Web user experience (WebQoE). As such, Google has proposed the concept of Above-the-Fold (ATF) to measure the time that the visible portion of the page takes to render. As in the example below, out of the 154 images in the page, only 8 are direclty visible ``above the fold'':  the time it takes to download and render these images (ATF=5.37s) is much lower than the time it takes to download all the data in the page (onLoad=16.11s). However the user will just notice the above the fold content in a first time.

Since measuring ATF time by inspecting the visual rendering of a Web page is costly, we have proposed and implemented very simple approximated version of the ATF concept. Checkout the illustrative YouTube video in the code section, or read the [PAM-18] paper for more information.

Time integral metrics

  • SpeedIndex
To better capture the visual progress of a Webpage, Google proposed the concept of SpeedIndex. Denoting with x(t) the normalized visual progress of a page, the SpeedIndex has an integral form as defined and illustrated below.

In a nutshell, if x(t) is the blue curve, then the SpeedIndex is the area X above this curve: intuitively, for two pages finishing the rendering process at the same onLoad time, the one whose rendering process starts early is preferable -- which will be reflected by a lower SpeedIndex value.

  • ByteIndex and ObjectIndex
While the SpeedIndex is an interesting concept, its first definitions required to capture snapshots of the video rendering process, makes it quite cumbersome to be used in pracice. As such, in [SIGCOMM-QoE-16] we proposed  two replacement metric, namely ObjectIndex and ByteIndex, that are structurally (and numerically) similar to the SpeedIndex but tremendously simpler to compute.

In a nutshell, we argue that the objects (or bytes) that are received by the browser (or the network card) can provide a good enough approximation of the visual completeness of the rendering process. We test SpeedIndex, ObjectIndex and ByteIndex (along with other metrics) on the Alexa top-100 dataset, finding high levels of correlation among these metrics as shown in the arc diagram above. For more details on the metrics, have a look at [SIGCOMM-QoE-16] or directly at the datasets we make available.

  • Integral cutoff at AATF

A final technicality is worth pointing out. While this may seem a tiny detail, Time Instant and Time Integral metrics can be combined. Particularly, going back to the SpeedIndex definition, observe that the integral can be terminated earlier than at the end of the page loading process, -- for instance, at the Above The Fold (ATF) time as illustrated below.

This improves WebQoE estimation, as we find in [SIGCOMM-QoE-16] over the datasets we make available, which we further discuss in the model section.