Information!

Here you can find the authors' copy of all our publications. If you use our code or datasets, we would ask you to please cite the appropriate reference. And even if you do not use our code or dataset but still want to cite our work, we will surely not stop you from doing so :)

  1. [TECHREP-19] Salutari, Flavia and Hora, Diego Da and Dubuc, Gilles and Rossi, Dario, Analyzing Wikipedia Users’ Perceived Quality Of Experience: A Large-Scale Study (Extended Technical Report) Technical Report dec. 2020, conference
    The Web is one of the most successful Internet application. Yet, the quality of Web users’ experience is still largely impenetrable. Whereas Web performances are typically gathered with controlled experiments, in this work we perform a large-scale study of one of the most popular websites, namely Wikipedia, explicitly asking (a small fraction of its) users for feedback on the browsing experience. The analysis of the collected users’ feedback reveals both expected (e.g., the impact of browser and network connectivity) and surprising findings (e.g., absence of day/night, weekday/weekend seasonality and other temporal dependencies) that we detail in this paper. Also, we leverage user survey responses to build supervised data-driven models to predict user satisfaction which, despite including state-of-the art quality of experience metrics, are still far from achieving accurate results. Finally, we make our dataset publicly available, which hopefully contributes in enriching and refining the scientific community knowledge on Web users’ Quality of Experience (QoE).
    @inproceedings{DR:TECHREP-19,
      author = {Salutari, Flavia and Hora, Diego Da and Dubuc, Gilles and Rossi, Dario},
      title = {Analyzing Wikipedia Users’ Perceived Quality Of Experience: A Large-Scale Study (Extended Technical Report)},
      booktitle = {Technical Report},
      month = dec,
      year = {2020},
      howpublished = {https://webqoe.telecom-paristech.fr/wikiqoe_extended_techrep.pdf}
    }
    
  2. [SIGCOMM-19] Huet, Alexis and Houidi, Zied Ben and Cai, Shengming and Shi, Hao and Xu, Jinchun and Rossi, Dario, Web Quality of Experience from Encrypted Packets ACM SIGCOMM Posters and Demos aug. 2019, DOI 10.1145/3342280.3342297 conference
    Pervasive encryption makes it hard for ISPs to manage their network. Yet, to avoid user churn at times of shrinking revenues, ISPs must be able to assess the quality of experience they are delivering to their customers. The case of the Web is particularly complex, with a plethora of recently proposed in-browser metrics that aim at capturing the page visual rendering quality (e.g. Above the Fold and SpeedIndex). In this demo, we showcase that such metrics can be estimated quite accurately just from streams of encrypted packets, using classic supervised learning techniques
    @inproceedings{DR:SIGCOMM-19,
      author = {Huet, Alexis and Houidi, Zied Ben and Cai, Shengming and Shi, Hao and Xu, Jinchun and Rossi, Dario},
      title = {Web Quality of Experience from Encrypted Packets},
      booktitle = {ACM SIGCOMM Posters and Demos},
      month = aug,
      year = {2019},
      doi = {10.1145/3342280.3342297},
      location = {Beijing, China},
      howpublished = {https://nonsns.github.io/paper/rossi19sigcomm.pdf}
    }
    
  3. [INFOCOM-19] Huet, Alexis and Rossi, Dario, Explaining Web users’ QoE with Factorization Machines IEEE INFOCOM apr. 2019, conference
    Whereas most of the literature employs classic machine learning techniques (such as C4.5 trees, Random Forest and Support Vector Machines) to improve forecast accuracy of QoE models, in this demo we explore the use of an information filtering system (Factorization Machine) to get fundamental insights and explain the relationship between QoE and different features.
    @inproceedings{DR:INFOCOM-19,
      author = {Huet, Alexis and Rossi, Dario},
      title = {Explaining Web users' QoE with Factorization Machines},
      booktitle = {IEEE INFOCOM},
      month = apr,
      year = {2019},
      location = {Paris, France},
      note = {keyword=qoe},
      howpublished = {https://nonsns.github.io/paper/rossi19infocom-b.pdf}
    }
    
  4. [WWW-19] Salutari, Flavia and Hora, Diego Da and Dubuc, Gilles and Rossi, Dario, A large-scale study of Wikipedia users’ quality of experience In proceedings of the 30th Web Conference (WWW’19) may. 2019, conference
    The Web is one of the most successful Internet application. Yet, the quality of Web users’ experience is still largely impenetrable. Whereas Web performances are typically gathered with controlled experiments, in this work we perform a large-scale study of one of the most popular Web sites, namely Wikipedia, explicitly asking (a small fraction of its) users for feedback on the browsing experience. We leverage user survey responses to build a data-driven model of user satisfaction which, despite including state-of-the art quality of experience metrics, is still far from achieving accurate results, and discuss directions to move forward. Finally, we aim at making our dataset publicly available, which hopefully contributes in enriching and refining the scientific community knowledge on Web users’ quality of experience (QoE).
    @inproceedings{DR:WWW-19,
      title = {A large-scale study of Wikipedia users’ quality of experience},
      author = {Salutari, Flavia and Hora, Diego Da and Dubuc, Gilles and Rossi, Dario},
      booktitle = {In proceedings of the 30th Web Conference (WWW'19)},
      address = {San Francisco, CA, USA},
      month = may,
      year = {2019},
      howpublished = {https://nonsns.github.io/paper/rossi19www.pdf}
    }
    
  5. [QoMEX-18] Hossfeld, Tobias and Metzger, Florian and Rossi, Dario, Speed Index: Relating the Industrial Standard for User Perceived Web Performance to Web QoE 10th International Conference on Quality of Multimedia Experience (QoMEX 2018) jun. 2018, conference
    In 2012, Google introduced the Speed Index (SI) metric to quantify the speed of the Web page visual completeness for the actually displayed above-the-fold (ATF) portion of a Web page. In Web browsing a page might appear to the user to be already fully rendered, even though further content may still be retrieved, resulting in the Page Load Time (PLT). This happens due to the browser progressively rendering all objects, part of which can also be located below the browser window’s current viewport. The SI metric (and variants) thereof have since established themselves as a de facto standard in Web page and browser testing. While SI is a step in the direction of including the user experience into Web metrics, the actual meaning of the metric and especially its relationship between Speed Index and Web QoE is however far from being clear. The contributions of this paper are thus to first develop an understanding of the SI based on a theoretical analysis and second, to analyze the interdependency between SI and MOS values from an existing public dataset. Specifically, our analysis is based on two well established models that map the user waiting time to a user ACR-rating of the QoE. The analysis show that ATF-based metrics are more appropriate than pure PLT as input to Web QoE models.
    @inproceedings{DR:QoMEX-18,
      author = {Hossfeld, Tobias and Metzger, Florian and Rossi, Dario},
      title = {Speed Index: Relating the Industrial Standard for User Perceived Web Performance to Web QoE},
      booktitle = {10th International Conference on Quality of Multimedia Experience (QoMEX 2018)},
      month = jun,
      year = {2018},
      location = {Sardinia, Italy},
      howpublished = {https://nonsns.github.io/paper/rossi18qomex.pdf}
    }
    
  6. [PAM-18] da Hora, Diego Neves and Asrese, Alemnew Sheferaw and Christophides, Vassilis and Teixeira, Renata and Rossi, Dario, Narrowing the gap between QoS metrics and Web QoE using Above-the-fold metrics International Conference on Passive and Active Network Measurement (PAM), Receipient of the Best dataset award mar. 2018, conference Award
    Page load time (PLT) is still the most common application Quality of Service (QoS) metric to estimate the Quality of Experience (QoE) of Web users. Yet, recent literature abounds with proposals for alternative metrics (e.g., Above The Fold, SpeedIndex and their variants) that aim at better estimating user QoE. The main purpose of this work is thus to thoroughly investigate a mapping between established and recently proposed objective metrics and user QoE. We obtain ground truth QoE via user experiments where we collect and analyze 3,400 Web accesses annotated with QoS metrics and explicit user ratings in a scale of 1 to 5, which we make available to the community. In particular, we contrast domain expert models (such as ITU-T and IQX) fed with a single QoS metric, to models trained using our ground-truth dataset over multiple QoS metrics as features. Results of our experiments show that, albeit very simple, expert models have a comparable accuracy to machine learning approaches. Furthermore, the model accuracy improves considerably when building per-page QoE models, which may raise scalability concerns as we discuss.
    @inproceedings{DR:PAM-18,
      title = {Narrowing the gap between QoS metrics and Web QoE using Above-the-fold metrics},
      author = {da Hora, Diego Neves and Asrese, Alemnew Sheferaw and Christophides, Vassilis and Teixeira, Renata and Rossi, Dario},
      booktitle = {International Conference on Passive and Active Network Measurement (PAM), Receipient of the Best dataset award},
      address = {Berlin, Germany},
      month = mar,
      year = {2018},
      note = {bestpaperaward},
      howpublished = {https://perso.telecom-paristech.fr/drossi/paper/rossi18pam-b.pdf}
    }
    
  7. [PAM-17] Bocchi, Enrico and De Cicco, Luca and Mellia, Marco and Rossi, Dario, The Web, the Users, and the MOS: Influence of HTTP/2 on User Experien ce Passive and Active Measurements apr. 2017, conference
    This work focuses on the evaluation of Web quality of experience as perceived by actual users and in particular on the impact of HTTP/1 vs HTTP/ 2. We adopt an experimental methodology that uses real web pages served through a realistic testbed where we control network, protocol, and application configur ation. Users are asked to browse such pages and provide their subjective feedbac k, which we leverage to obtain the Mean Opinion Score (MOS), while the testbed r ecords objective metrics. The collected dataset comprises over 4,000 grades that we explore to tackle the question whether HTTP/2 improves users experience, to what extent, and in which conditions. Findings show that users report marginal d ifferences, with 22%, 52%, 26% of HTTP/2 MOS being better, identical, or wors e than HTTP/1, respectively. Even in scenarios that favor HTTP/2, results are no t as sharp as expected. This is in contrast with objective metrics, which instea d record a positive impact with HTTP/2 usage. This shows the complexity of under standing the web experience and the need to involve actual users in the quality assessment process.
    @inproceedings{DR:PAM-17,
      title = {The Web, the Users, and the MOS: Influence of HTTP/2 on User Experien
      ce},
      author = {Bocchi, Enrico and De Cicco, Luca and Mellia, Marco and Rossi, Dario},
      year = {2017},
      month = apr,
      booktitle = {Passive and Active Measurements},
      halid = {hal-01613491},
      pages = {},
      howpublished = {https://perso.telecom-paristech.fr/drossi/paper/rossi17pam.pdf}
    }
    
  8. [SIGCOMM-QoE-16] Bocchi, Enrico and De Cicco, Luca and Rossi, Dario, Measuring the Quality of Experience of Web users ACM SIGCOMM Workshop on QoE-based Analysis and Management of Data Communication Networks (Internet-QoE 2016), selected as best paper in the workshop for reprint in ACM SIGCOMM Comput. Commun. Rev. aug. 2016, conference Award
    @inproceedings{DR:SIGCOMM-QoE-16,
      title = {Measuring the Quality of Experience of Web users},
      author = {Bocchi, Enrico and De Cicco, Luca and Rossi, Dario},
      year = {2016},
      month = aug,
      booktitle = {ACM SIGCOMM Workshop on QoE-based Analysis and Management of Data Communication Networks (Internet-QoE 2016), selected as best paper in the workshop for reprint in ACM SIGCOMM Comput. Commun. Rev.},
      note = {bestpaperaward},
      howpublished = {https://perso.telecom-paristech.fr/drossi/paper/rossi16internet-qoe.pdf}
    }
    
  9. [DIRECTORSCUT-16] Bocchi, Enrico and De Cicco, Luca and Mellia, Marco and Rossi, Dario, HTTP/2 vs the Users: The Good, The Bad and The Ugly (Director’s Cut) Technical Report apr. 2016, conference
    HTTP/2 is reality: services adopt it and researchers measure and optimize performance – yet, the actual impact on users’ web browsing experience is unclear. This work focuses on the comparison of HTTP/1.1 and HTTP/2 as perceived by the user. We adopt an experimental methodology, where popular web pages are served through a testbed that allows us to control network, protocol and application configuration. We ask users to browse actual web pages and provide their subjective feedback, i.e., the Mean Opinion Score (MOS), while we record objective metrics. We collect a dataset, that will be made available to the community, accounting for several thousands MOS grades, and leverage it to tackle the question whether HTTP/2 is better than HTTP/1.1 as far as user experience is concerned. We find that, despite differences can be seen from objective metrics, (i) users report much smaller differences. Additionally, we show how (ii) MOS is poorly correlated with any objective metric, to the point that (iii) accurately predicting MOS from objective metrics is a real challenge. At last, (iv) when trying to correlate how different factors (e.g., network configuration, page characteristics, etc.) contribute to the quality of experience, we observe much less clear results than when observing objective metrics.
    @inproceedings{DR:DIRECTORSCUT-16,
      title = {HTTP/2 vs the Users: The Good, The Bad and The Ugly (Director's Cut)},
      author = {Bocchi, Enrico and De Cicco, Luca and Mellia, Marco and Rossi, Dario},
      year = {2016},
      month = apr,
      booktitle = {Technical Report},
      howpublished = {https://perso.telecom-paristech.fr/drossi/paper/director.pdf},
      pages = {}
    }
    
  10. [INFOCOM-IC-16] Bocchi, Enrico and De Cicco, Luca and Rossi, Dario, Web QoE: Moving beyond Google’s SpeedIndex Finalist at the IEEE INFOCOM Innovation Challenge, apr. 2016, conference
    The World Wide Web is still among the most prominent Internet applications. While the Web landscape has been in perpetual movement since the very beginning, these last few years have witnessed some noteworthy proposals such as SPDY, HTTP/2 and QUIC which could disrupt the Web status quo and profoundly reshape the protocols family at application layer. Technically solid means are clearly needed to assess whether these new protocols can keep their promises: The risk is that these new protocols could otherwise fail to be adopted. While this investigation is already under way, both the industry and the research community are in our opinion expressing the right question, to which they however answer using the wrong tools. Over the years, webpages have grown to quite complex entities including hundreds of objects of several types, sharded over many domains. Yet, the current practice is to express Web Quality of Experience (QoE) via the document completion time (onLoad) despite its known inaccuracy and poor correlation with the actual user experience. At the same time, while better metrics do exist (e.g., the SpeedIndex, proposed by Google in 2012), they are complex to evaluate and require a prohibitive amount of computing resources (i.e., record filmstrips of the visual rendering in the browser). As such, their use is limited to lab experiments, but have to date failed to catch up in larger scale. Fortunately, there is a way out of this impasse: as it often happens, once found, the solution is very simple and elegant.
    @inproceedings{DR:INFOCOM-IC-16,
      title = {Web QoE: Moving beyond Google's SpeedIndex},
      author = {Bocchi, Enrico and De Cicco, Luca and Rossi, Dario},
      year = {2016},
      month = apr,
      booktitle = {Finalist at the IEEE INFOCOM Innovation Challenge,},
      pages = {},
      howpublished = {https://perso.telecom-paristech.fr/drossi/paper/rossi16infocom-innovation-challenge.pdf}
    }
    
  11. [GLOBECOM-03] Rossi, D. and Casetti, C. and Mellia, M., User Patience and the Web: a Hands-on Investigation IEEE Globecom’03 dec. 2003, conference
    We present a study of web user behavior when network performance decreases causing the increase of page transfer times. Real traffic measurements are analyzed to infer whether worsening network conditions translate into greater impatience by the user, which translates in early interruption of TCP connections. Several parameters are studied, to gather their impact on the interruption probability upon web transfers: times of day, file size, throughput and time elapsed since the beginning of the download. Results presented try to paint a picture of the complex interactions between user perception of the Web and network-level events.
    @inproceedings{DR:GLOBECOM-03,
      author = {Rossi, D. and Casetti, C. and Mellia, M.},
      title = {User Patience and the Web: a Hands-on Investigation},
      booktitle = {IEEE Globecom'03},
      address = {San Francisco, CA, USA},
      year = {2003},
      month = dec,
      howpublished = {https://perso.telecom-paristech.fr/drossi/paper/rossi03globecom.pdf}
    }