DHAP
From MAGGIE
Contents |
[edit] Project Title
Detecting hosts with anomalous performance compared to neighbors
[edit] Project Aim
The aim of this project is to assist in detecting hosts that have anomalous network performance compared to hosts that are "nearby".
[edit] Motivation
Currently PingER monitors about 600 hosts in over 125 countries from 30+ monitoring sites in 15 countries around the world. All these corresponds to about 100,000 measurements per day (or ~2,000,000 pings/day) between over 2200 monitor site/remote site pairs. To provide summary information the data can be aggregated by country, by region (e.g. Middle East, Africa), or sub-region (e.g. Central Africa, Mediterranean). Ideally all the hosts in a an aggregation should behave similarly and in the ideal case one would only need one host per aggregation to characterize the aggregation. However, this is not the case. For example, some the paths to the monitoring host from some remote-hosts in a region may be via geo-stationary satellite links (with over 600ms minimum Round-Trip_Times (RTTs)), while others may have land-lines with much lower RTTs, some sites maybe very isolated with only wireless or a slow speed shared line, while another nearby but in a city may have excellent connectivity. Thus when characterizing some metric (e.g. loss, RTT) for an aggregation we not only need the mean or median but also some idea about the distribution or measure of the variability. In some case large variability may indicate bad data:
- for example a host that purports (e.g. by its country code) to be in say Namibia is actually located in Europe (e.g. often developing countries have web proxies in countries with better connectivity).
- a host may be poorly managed, or have a poorly managed connection to the outside world.
Identifying such bad data will enable us to modify the PingER host configuration database (NODEDETAILS) to correctly locate hosts, or to provide information to the host/site contacts so they can investigate possible problems, and seek improvements.
[edit] Project Description
Some work has already been done on this by Waqar Ali, see PingER Anomalous Sites Within the Last Week, displaying the anomalous (i.e. those with >2 standard deviations from other hosts in the same region) minimum RTTs. This project will build on and extend this work as follows:
- Provide anomalous detection assistance for other metrics, in particular: loss, jitter, derived TCP throughput, average RTT, unreachability;
- Provide tables for country aggregations as well as regions;
- Improve the navigation to other relevant data, e.g. access to tables showing metric details of all the hosts in the aggregation;
- Improved visualization to show distributions more clearly, e.g. using Smokeping or Box plots.
The second stage of the project will be to use the tools developed to study and characterize the variability of Internet performance by region/countries. This may also lead to the development of extra tools.
[edit] Requirements
- The student will need to be or become proficient in Unix/Linux, perl and CGI scripts.
- The student will also need to become proficient in the basic statistics of distributions and analysis techniques.
- The code will need to be production quality. Guidelines on how to write perl will be provided (see IEPM Perl Coding Style and Coding Style).
- The student will need to apply for and get a Unix account at SLAC and will be provided access to the relevant computers, files and databases. (contact Umar Kalim for details).
