Linus on Evolution

Here is a great rant from Linus Torvalds back in 2002 on biological and software evolution.
... from a thread on LKML, summarized here: http://kerneltrap.org/node/11

---

You know what the most complex piece of engineering known to man in the whole solar system is?

Guess what - it's not Linux, it's not Solaris, and it's not your car.

It's you. And me.

And think about how you and me actually came about - not through any complex design.

Right. "sheer luck".

Well, sheer luck, AND:

• free availability and _crosspollination_ through sharing of "source code", although biologists call it DNA.
• a rather unforgiving user environment, that happily replaces bad versions of us with better working versions and thus culls the herd (biologists often call this "survival of the fittest")
• massive undirected parallel development ("trial and error")

I'm deadly serious: we humans have _never_ been able to replicate something more complicated than what we ourselves are, yet natural selection did it without even thinking.

Don't underestimate the power of survival of the fittest.

And don't EVER make the mistake that you can design something better than what you get from ruthless massively parallel trial-and-error with a feedback cycle. That's giving your intelligence _much_ too much credit.

Linus

Python - URL Timer - Web Monitor Utility

In a previous post, I showed how to insert timers between steps in an http request to do some basic profiling. Peter Bengtsson contacted me with some enhancements to the original code. I took his enhancements and added some more of my own to create a neat little web monitoring utility.

It uses Python's httplib to send a GET requests at regular intervals to a URL and adds some timing instrumentation between each step of the request/response. Results are printed to the console.

get it here:
http://code.google.com/p/corey-projects/source/browse/trunk/python2/url_timer.py

to use, give it a site and interval:

> python url_timer.py www.goldb.org 5

... or an https/ssl url and an interval:

> python url_timer.py https://ssl.sonic.net/ 5

sample output:

>python url_timer.py www.goldb.org 5
request sent         response received    content transferred  size
------------         -----------------    -------------------  ----
0.0040 (0.0040)      0.2830 (0.2830)      0.2846 (0.2846)      9810 bytes (0.010 MB total)
0.0009 (0.0024)      0.2790 (0.2810)      0.2803 (0.2825)      9810 bytes (0.020 MB total)
0.0009 (0.0019)      0.2841 (0.2821)      0.2854 (0.2834)      9810 bytes (0.029 MB total)
0.0010 (0.0017)      0.2775 (0.2809)      0.2789 (0.2823)      9810 bytes (0.039 MB total)
0.0008 (0.0015)      0.2794 (0.2806)      0.2820 (0.2822)      9810 bytes (0.049 MB total)
0.0009 (0.0014)      0.2845 (0.2813)      0.2859 (0.2828)      9810 bytes (0.059 MB total)
0.0010 (0.0013)      0.2796 (0.2810)      0.2809 (0.2826)      9810 bytes (0.069 MB total)
0.0009 (0.0013)      0.2798 (0.2809)      0.2808 (0.2823)      9810 bytes (0.078 MB total)
0.0009 (0.0012)      0.2791 (0.2807)      0.2803 (0.2821)      9810 bytes (0.088 MB total)
...

Automated Web/HTTP Profiler with Selenium-RC and Python

A small new open source project I am working on:

Selenium-profiler is a web/http profiler built with Selenium-RC and Python. It profiles page load time and network traffic for a web page. The profiler uses Selenium-RC to automate site navigation (via browser), proxy traffic, and sniff the proxy for network traffic stats as requests pass through during a page load.

It is useful to answer questions like:

• how many http requests does that page make?
• how fast are the http responses coming back?
• which http status codes are returned?
• how many of each object type are requested?
• what is the total page load time?

Get it here:
http://selenium-profiler.googlecode.com/

Sample Output:

--------------------------------
results for http://www.google.com/

content size: 38.06 kb

http requests: 9
status 200: 6
status 204: 1
status 403: 2

profiler timing:
2.063 secs (page load)
2.047 secs (network: end last request)
0.111 secs (network: end first request)

file extensions: (count, size)
gif: 1, 8.558 kb
ico: 2, 2.488 kb
js: 2, 18.046 kb
png: 1, 5.401 kb
unknown: 3, 3.567 kb

http timing detail:
403, GET, /favicon.ico, 111 ms
200, GET, /newkey, 738 ms
200, GET, /, 332 ms
403, GET, /favicon.ico, 3 ms
200, GET, /logo.gif, 255 ms
200, GET, /d9n_Nh4I09g.js, 411 ms
200, GET, /2cca7b2e99206b9c.js, 65 ms
204, GET, /generate_204, 88 ms
200, GET, /nav_logo7.png, 49 ms

Selenium RC with Python in 30 Seconds

Selenium is a suite of tools to automate web app testing across many platforms. It has various pieces (Core, RC, IDE, etc), and I struggled trying to figure out how everything fits together and works. At the end of the day, all I wanted to do was use Selenium from my Python code to drive a browser session.

Selenium ships with full tests and some good sample code, but the driver examples all contain test frameworks/runners (JUnit, unittest, etc). All I wanted was a simple way to integrate with Python. To do this, the piece I need is Selenium RC (which includes Selenium Core).

So here is the beginners' 30-second guide to getting the Python client driver working with Selenium-RC:

1. Install Python (and add it to your path)
2. Install Java (and add it to your path)
3. Download Selenium RC
4. Unzip Selenium RC and search for 'selenium-server.jar' and 'selenium.py'
5. Copy them to a directory and run 'java -jar selenium-server.jar' to start the server
6. Start writing your Python driver code! (import selenium)

So what's actually going on when you run a driver script?

Selenium-server is the core program, which also contains an integrated web server. You send HTTP requests to the server to instruct it how to drive your browser. The 'selenium.py' module is just a wrapper around 'httplib' that provides a Python API for interacting with the Selenium-server. You need to import the 'selenium.py' module from your script and then you are ready to go.

now let's test it out and see if it works. Try the following Python script. It should open Firefox and navigate to www.google.com.

#!/usr/bin/env python

from selenium import selenium

sel = selenium('localhost', 4444, '*firefox', 'http://www.google.com/')
sel.start()
sel.open('/')
sel.wait_for_page_to_load(10000)
sel.stop()

Python - HTTP Request Profiler

Here is a script in Python that profiles an HTTP request to a web server.

to use, give it a url:

> python http_profiler.py www.goldb.org

output:

0.12454 request sent
0.24967 response received
0.49155 data transferred (9589 bytes)

It uses Python's httplib to send a GET request to the URL and adds some timing instrumentation between each step of the request/response.

Python Code:

#!/usr/bin/env python
# Corey Goldberg - September 2009


import httplib
import sys
import time



if len(sys.argv) != 2:
    print 'usage:\nhttp_profiler.py \n(do not include http://)'
    sys.exit(1)

# get host and path names from url
location = sys.argv[1]
if '/' in location:
    parts = location.split('/')
    host = parts[0]
    path = '/' + '/'.join(parts[1:])
else:
    host = location
    path = '/'

# select most accurate timer based on platform
if sys.platform.startswith('win'):
    default_timer = time.clock
else:
    default_timer = time.time

# profiled http request
conn = httplib.HTTPConnection(host)
start = default_timer()  
conn.request('GET', path)
request_time = default_timer()
resp = conn.getresponse()
response_time = default_timer()
size = len(resp.read())
conn.close()     
transfer_time = default_timer()

# output
print '%.5f request sent' % (request_time - start)
print '%.5f response received' % (response_time - start)
print '%.5f content transferred (%i bytes)' % ((transfer_time - start), size)

Web Page Profilers for Windows/Internet Explorer

A "Web Page Profiler" is a tool that is used to measure and analyze web page performance. It is usually implemented as a browser plugin or add-on, and lets you see performance of web pages/objects as they are transferred/loaded/executed/rendered.

For Firefox, choosing a profiler is a no-brainer. Firebug is an excellent developer tool that includes profiling capabilities.

• Firebug

Unfortunately, Firebug does not work with Microsoft's Internet Explorer. There is Firebug Lite, which "can" work with IE, but it is limited in functionality and requires you to install some server side code (which is not always feasible).

• Firebug Lite

There are some web profilers specifically for IE, but none of them live up to the functionality or stability of Firebug. Some to look at are:

• AOL Pagetest
• MySpace Performance Tracker

... also, Internet Explorer 8 has built-in "Developer Tools" (press F12) that include a basic performance profiler for JavaScript execution.

Does anyone know of any other profilers/performance tools that work with IE? Thoughts?

Segue/Borland/MicroFocus SilkPerformer - Who Owns It?

SilkPerformer is a popular performance and load testing tool. It has a confusing history and has bounced around between companies quite a bit. Here is where it came from and the current state of who owns it.

The history AFAIK:

• Segue buys a product from some (scandanavian?) company and names it Segue SilkPerformer.
• Borland buys Segue and names it Borland SilkPerformer.
• Micro Focus buys Borland and name it Micro Focus SilkPerformer.

Right now it is a little confusing because Borland still lists it as a product. The acquisition of Borland just went through a few days ago and now Micro Focus has it listed on its site also. I would expect the Borland site to eventually disappear.

To make it even more confusing; Micro Focus also bought Compuware which had its own performance/load testing application. So now Micro Focus is marketing QALoad as well as SilkPerformer, which overlap in most functionality.

It will be interesting to see where this goes and what happens to the product lines.

[Software] Tools Don't Make You a Mechanic

FYI: Learning LoadRunner does not mean that one has learned Performance Testing. LoadRunner is only a tool. Learning to use one of these:

...does not mean that one can disassemble and reassemble one of these:

[via JakeBrake]