Models at a glance
Roughly, there are two classes of models that can be used to approximate subjective user QoE from objective metrics. Our findings are reported in [QoMEX-18] and [PAM-18] and are briefly summarized here.
On the one hand, there are expert models, where domain experts specify a closed form function on (typically few) variables, and use subjective MOS data to fit model parameters. On the other hand, there are data-driven models where machine learning experts employ MOS data to train the model based on (typically multiple) collected features.
Expert models
Two well established expert models of Web QoE are the ITU-T model (that follows the Weber-Fechner Law and assumes that the user QoE has a logarithmic relationship with the underlying QoS metric), and the IQX model (that postulates an exponential interdependency between QoE and QoS).
We have thoroughly analyzed these models under the light of our new datasets in [QoMEX-18] and [PAM-18]. Shortly, we find that these models are still valid to some extent, and that their accuracy greatly improves when the model employs one of our advanced metrics (particularly, the ImageIndex with integral cutoff at the ATF time… but to understand this you really have to go through our papers) instead of the classic Page Load Time (PLT).
Data-driven models
Supervised data-driven models can be learned from the data using one of the many options available nowaday. Particularly, the problem of guessing subjective user grades from objective measurement can be phrased as a regression (using MOS as a continuous variable), classification (quantizing MOS in few discrete values) or reccomendation (targeting individual user grades) problems.
In [PAM-18], we leverage regression techniques such as SVR, CART and BOOST, using from as few as 3 to as many as 27 features, to forecast MOS – finding that the improvement they bring over expert models is modest in practice.
Then, in [WWW-19] we leverage classification techniques such as XGBoost and Random Forests to forecast whether the users are satisfied or not with the rendering of Wikipedia pages – a binary classification problem that is surprisingly hard even with state of the art objective features.
Finally, we resort to recommender systems such as Factorization Machines to get fundamental insights and explain the relationship between QoE and different observables. FM is particularly suited in our case as it allows estimation of parameters under very sparse data, without loosing information by aggregating samples into mean scores, so that we can study both the user and the webpage angles – finding that they bring as much information as the objective metrics. Check out our interactive demo !