{{{
#!forumlinks
}}}

= Log File Analysis =

Steven Koolen created numerous scripts to 
parse crawling results on Tribler network.

Everything is stored on 
SuperStorage3:/home/steven/permid_mrt-nov

Both Data, .gnuplot, and .py

1. get all raw crawl data files. [[BR]]
2. run tribler_log.py, this creates .ips files[[BR]]
3. run tribler_log_duplicate.py, which filters duplicate entries (might choose to also run tribler_log_filter_multipleinstance.py in order to filter back and forth switchers, thus using multipe computers)[[BR]]
4. run any of the tribler_log_*.py files to create graphs from the data[[BR]]
    For example tribler_log_permids_day.py shows the number of users [[BR]]
 
[[BR]]
tribler_log_runall.sh runs all files for step 4 and creates .eps graph files

{{{
#!protected
#:[[Include(wiki:ProtectedSectionMessage)]]


= (Super)peer Experiments =

On 16 March 2007 BuddyCast2 was taking into production
use on 1 process of Superpeer1. On 22 March all superpeer
processes where upgraded.

From 6 March until 22 March a torrentcollecting experiment
was run on SP1 port 7002, not a superpeer.txt entry.

The log files can be found on Superstorage3:/home/tribler/backup/superpeer2.experiment.log.tar.gz

= ToDo list for the Summer =

Goal is to :
''Create tools which provide real-time insight (10 minutes update interval)
into the performance of the Buddycast epidemic protocol
for peer discovery and content distribution''.

  - Try to replicate the work of Steven
  - Create Gnuplot graphs of the historic Superpeer logs
    * Number of peers in time
    * Newly installed peers per day
  - Expand with recent log files of Superpeers
  - Expand scripts with logs of multiple Buddycast peers (Superstorage1-3, ask Jie)
  - Create complete toolchain for real-time analysis
    * Copy new logs from Superpeers
    * run script with historic + new data
    * Try to mimic the [http://awstats.sourceforge.net/ awstats type of interface]

All previous data and code can be reached through :

ssh jantet@superstorage2:/home/jantet/previous_work

= Results =

The program is composed by two groups of python scripts:
 - tribler_log2data read all log file of the directory given in parameter, and create three plain-text databases over Permid, Connection, and Hours. This script remember the name of the last scaned log file, to start it next scan at this position.
 - The other scripts: tribler_datam2graph, tribler_datah2graph and tribler_datad2graph use databases from the first program to create data file which can be drawn by gnuplot.

Currents graphs can be viewed [http://www.pds.ewi.tudelft.nl/~jantet/ here]
They came from Superpeer-1-Process-1's log files.

Currents graphs are not easy to read, because they contains too many informations. We have to choose which informations are pertinent, and change the gnu script file to display only them.

Create complete toolchain is not yet possible because the server is running over Windows, and Gnuplot is not install on. For the moment, only python scripts can be shell from a web page.

In the "graph/" directory, there is some shell scripts which create a png image from a .dat file.
For exemple: "sh make_graph_month.sh 2006 10" create the tribler_graph_month.png image for the month of october 2006

The full HowTo documentation will be [http://www.pds.ewi.tudelft.nl/~jantet/howto.html here]

== Statistics ==

The most interesting statistics would be :
  - How many new Tribler users: New PermIDs in (hour,day,month,year,overall)
  - How many users we have : Total active PermIDs per (hour,day,month,year,overall)
  - Exactly much they use Tribler : PermID activity per (hour,day,month,year,overall)
    * Just 1 connection in (hour,day,month,year,overall)
    * Just 2 connections
    * Just 3 connections
    * Just 4 connections
    * Between 5-10 connections
    * Between 10-25 connections
    * Between 25-100 connections
    * Over 100+ connections

The "just 1 connection" statistics are of particular interest. They
show the amount of ''one-minute users''. People who visited Tribler.org, downloaded the
software, got it running, and then decided never to use it again.
A graph which shows of the past year how many people only use Tribler
once is an important metric of overall success and appeal.


}}}