Python - Linux: Parse Network Stats From ifconfig

I needed to get some Linux networking stats in my Python program today. Specifically, I needed 'bytes sent' and 'bytes received' counts since last reboot from the local machine.

ifconfig is a network configuration utility for Linux that you run from the command line:

corey@studio17:~$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:22:19:e5:07:31  
          inet addr:10.0.0.5  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::222:19ff:fee5:731/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3353822 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3052408 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3476310326 (3.4 GB)  TX bytes:256706611 (256.7 MB)
          Interrupt:17 

The following function parses output from ifconfig to get the network stats I was after:

import re
import subprocess

def get_network_bytes(interface):
    output = subprocess.Popen(['ifconfig', interface], stdout=subprocess.PIPE).communicate()[0]
    rx_bytes = re.findall('RX bytes:([0-9]*) ', output)[0]
    tx_bytes = re.findall('TX bytes:([0-9]*) ', output)[0]
    return (rx_bytes, tx_bytes)

Example usage:

import re
import subprocess

def main():
    rx_bytes, tx_bytes = get_network_bytes('eth0')
    print '%s bytes received' % rx_bytes
    print '%s bytes sent' % tx_bytes
      
def get_network_bytes(interface):
    output = subprocess.Popen(['ifconfig', interface], stdout=subprocess.PIPE).communicate()[0]
    rx_bytes = re.findall('RX bytes:([0-9]*) ', output)[0]
    tx_bytes = re.findall('TX bytes:([0-9]*) ', output)[0]
    return (rx_bytes, tx_bytes)

if __name__ == '__main__':
    main()

Update: someone left an anonymous comment and mentioned you can just read from proc/net/dev rather than using ifconfig. I modified his code sample and came up with this:

def get_network_bytes(interface):
    for line in open('/proc/net/dev', 'r'):
        if interface in line:
            data = line.split('%s:' % interface)[1].split()
            rx_bytes, tx_bytes = (data[0], data[8])
            return (rx_bytes, tx_bytes)

Example Usage:

def main():
    rx_bytes, tx_bytes = get_network_bytes('eth0')
    print '%s bytes received' % rx_bytes
    print '%s bytes sent' % tx_bytes
      
def get_network_bytes(interface):
    for line in open('/proc/net/dev', 'r'):
        if interface in line:
            data = line.split('%s:' % interface)[1].split()
            rx_bytes, tx_bytes = (data[0], data[8])
            return (rx_bytes, tx_bytes)

if __name__ == '__main__':
    main()

My Chrome Users Now Outnumber Internet Explorer (Traffic Stats)

I really like to analyze the traffic to my various websites (homepage, blog, project pages, open source code).

Since my visitors must be somewhat interested in what I post, I like to think they are *just* *like* *me*.

So... of course, my own personal web metrics are better than anything else out there :)
(to you they might be worthless, but to me they are gold)

I get a pretty steady stream of about 40k pageviews per month. In the past year, I've served a paltry half million pageviews. But hey, its a good enough sample size to work with, and pretty flattering to have attracted that many eyeballs.

Look what happened this past month:

(stats from aug_22_2010 - sept_21_2010)

Chrome visitors overtook Internet Explorer in site usage.

Here is how it has been playing out in the past year and a half:

(Notice my Firefox visitors are waning a little also.)

Python - Multi-Mechanize Script With HTTP Profiling Timers

Multi-Mechanize is an open source framework for web performance and load testing. It allows you to run simultaneous python scripts to generate load (synthetic transactions) against a web site or web service.

The default response timer wraps the entire Transaction(), so it will time everything included. For more granular timing, you need to instrument your script code with custom timers.

Multi-Mechanize is pure Python and you have access to all of Python's standard library in your scripts. For example, you can use httplib to write a virtual user agent script and get detailed HTTP profiling times (TTFB, TTLB, etc). http://docs.python.org/library/httplib.html

 
import httplib
import time


class Transaction(object):
    def __init__(self):
        self.custom_timers = {}
    
    def run(self):
        conn = httplib.HTTPConnection('www.example.com')
        start = time.time()
        conn.request('GET', '/')
        request_time = time.time()
        resp = conn.getresponse()
        response_time = time.time()
        conn.close()     
        transfer_time = time.time()
        
        self.custom_timers['request sent'] = request_time - start
        self.custom_timers['response received'] = response_time - start
        self.custom_timers['content transferred'] = transfer_time - start
        
        assert (resp.status == 200), 'Bad HTTP Response'

To test it out, you can add this to the bottom of the script and run it from the command line:

 
if __name__ == '__main__':
    trans = Transaction()
    trans.run()
    
    for timer in ('request sent', 'response received', 'content transferred'):
        print '%s: %.5f secs' % (timer, trans.custom_timers[timer])

Output:

 
request sent: 0.14429 secs
response received: 0.25995 secs
content transferred: 0.26007 secs

* if you are running MS Windows, replace the time.time() calls with time.clock() for better timer accuracy. On all other operating systems, use time.time()

CSharp/.NET - Use Stopwatch() Instead of DateTime() for Accurate High Precision Timing

(originally posted at coreygoldberg.blogspot.com)

In C# (.NET), the DateTime() class is not accurate for high precision timing. Instead, use the Stopwatch() class if you need a timer. Most hardware and operating systems support a high-resolution performance counter that can be accessed through System.Diagnostics.Stopwatch.

Don't use DateTime() like this if you need accuracy:

 
using System;

class Program
{
    public static void Main()
    {
        DateTime start = DateTime.Now;

            // do timed work here

        DateTime stop = DateTime.Now;

        // don't do this. you won't get accurate timing
        Console.WriteLine("{0} ms", (stop - start).TotalMilliseconds);

        // definitely don't do this. you won't get accurate timing or full timer resolution
        Console.WriteLine("{0} ms", (stop - start).Milliseconds);
    }
}

Stopwatch() uses operating system's high-resolution performance counter:

 
using System;
using System.Diagnostics;

class Program
{
    public static void Main()
    {
        Stopwatch stopWatch = Stopwatch.StartNew();

            // do timed work here

        stopWatch.Stop();

        // don't do this. you won't get full timer resolution
        Console.WriteLine("{0} ms", stopWatch.ElapsedMilliseconds);

        // do this to get accurate high precision timing
        Console.WriteLine("{0} ms", stopWatch.Elapsed.TotalMilliseconds);
    }
}

The Stopwatch class is in the System.Diagnostics namespace:

 
using System.Diagnostics;

Stopwatch measures elapsed time by counting timer ticks in the underlying timer mechanism. If the installed hardware and operating system support a high-resolution performance counter, then the Stopwatch class uses that counter to measure elapsed time. Otherwise, the Stopwatch class uses the system timer (DateTime class) to measure elapsed time.

To see if your system supports a high-resolution performance counter, check the Stopwatch.IsHighResolution property:

 
if (Stopwatch.IsHighResolution)
    Console.WriteLine("Using the system's high-resolution performance counter.");
else 
    Console.WriteLine("Using the DateTime class.");

To check the timing accuracy, use the Stopwatch.Frequency property:

 
long frequency = Stopwatch.Frequency;

Console.WriteLine("Timer frequency in ticks per second: {0}", frequency);

long nanosecPerTick = (1000L*1000L*1000L) / frequency;

Console.WriteLine("Timer is accurate within {0} nanoseconds", nanosecPerTick);