Over at "Inside the Nerdery" (love the name), Jodi Chromey did a short writeup on web performance and analysis tools. Pylot made the list :)
6 tools you can use to analyze your Web site’s performance
"Everyone wants a Web site that’s running at optimum speed, but for a lot of people they have no idea what the heck that is. That’s why the Nerds have created this list of six Web performance analysis tools."
I've been writing code in Python 3.0 (3000) this week. It feels really nice. Basically the same old Python, with some corners cleaned up and reorganized.
I have tons of legacy code to support that is written in 2.4/2.5, so I still need to keep my hands in Python 2.x for quite a long time. But when will I switch over all new projects so they use Python 3.0?
Well the answer is: as soon as my 5 favorite libraries work with Python 3.0. Here is what I'm waiting on:
I have no idea when these will be ported to work with 3.0. Hopefully I'm not waiting *years* for this.
When are you switching or migrating?
My monitoring tool: WebInject is a popular plugin in the Nagios community. A Nagios user (Felipe Ferreira) just wrote a small how-to on setting up WebInject. (though the official manual is much more thorough)
Webinject How to
"When monitoring websites, many times we will need more than just checking if the site is up. We may need to see if the internals of the website are working. A good example is making a user login check. Using Nagios alone it can not be done, but thanks to Corey Goldberg, it is possible using his script webinject.pl. [snip...] And yes it works with HTTPS and redirections."
Thanks Felipe!
Here is a Python script to reboot a remote Windows server. It requires the Python Win32 Extensions to be installed.
# Reboot a remove server
import win32security
import win32api
import sys
import time
from ntsecuritycon import *
def RebootServer(message='Server Rebooting', timeout=30, bForce=0, bReboot=1):
AdjustPrivilege(SE_SHUTDOWN_NAME)
try:
win32api.InitiateSystemShutdown(None, message, timeout, bForce, bReboot)
finally:
# Remove the privilege we just added.
AdjustPrivilege(SE_SHUTDOWN_NAME, 0)
def AbortReboot():
AdjustPrivilege(SE_SHUTDOWN_NAME)
try:
win32api.AbortSystemShutdown(None)
finally:
# Remove the privilege we just added.
AdjustPrivilege(SE_SHUTDOWN_NAME, 0)
def AdjustPrivilege(priv, enable=1):
# Get the process token.
flags = TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY
htoken = win32security.OpenProcessToken(win32api.GetCurrentProcess(), flags)
# Get the ID for the system shutdown privilege.
id = win32security.LookupPrivilegeValue(None, priv)
# Obtain the privilege for this process.
# Create a list of the privileges to be added.
if enable:
newPrivileges = [(id, SE_PRIVILEGE_ENABLED)]
else:
newPrivileges = [(id, 0)]
# and make the adjustment.
win32security.AdjustTokenPrivileges(htoken, 0, newPrivileges)
if __name__=='__main__':
message = 'This server is pretending to reboot\nThe shutdown will stop in 10 seconds'
RebootServer(message)
print 'Sleeping for 10 seconds'
time.sleep(10)
print 'Aborting shutdown'
AbortReboot()
Note: This code came from somewhere on the net and was not attributed to anyone. If you wrote it, let me know.
Holiday reading just arrived...
Finally got my hands on the brand new "Programming in Python 3 - A Complete Introduction to the Python Language". As far as I know, this is the first print book covering Python 3.0 (Python 3000). A quick skim looked promising.
Update: I've read a good chunk of this book now, and it is excellent. It's very clearly written and all code examples are clean and follow nice conventions. It covers a wide variety of topics including all the important stuff in the language and standard library. This book would make a good mid level introductory book to the language, or as a good reference/refresher to a more experience Python hacker. I highly recommend picking this up.
I've finally realized...
REST is a godsend for Performance and Load Testing. It's not just a steaming pile of crap tunneled over HTTP.
Dropbox is a file synching and backup tool that I began using a few months back. It is now one of my favorite tools.
You install a service and designate a folder as your "dropbox" and then register the machine with your online account. You can do this on as many machines as you want. When you add content to the folder, all the registered machines with it installed synch the folder together.
So I have one common "code" folder on my 2 home laptops and work PC that I keep in synch. It also stores the data in a secure web repository and has version control.
Dropbox is useful for synching multiple machines or collaborating with a group of people that have synched dropboxes to the same account. It works over the web on port 80 so you don't have to worry about VPN's or corporate firewalls or anything. It also works on almost all platforms and is free (up to 2GB storage).
Wordle generates "word clouds" from text that you provide. Here is a Wordle mashup showing the word density in my Tweets from Twitter:
In a recent blog post, I introduced a script for remotely monitoring Windows machines. It is used to retrieve performance/health metrics. It uses WMI (Windows Management Instrumentation) to interact with the Windows machine and its Win32 classes.
Now I will show how you an example of using the module (which I named "machine_stats.py"). I call it from a multithreaded controller to poll multiple machines in parallel.
The script below reads a list of Host Names (or IP's) from a file named "machines.txt". It then spawns a thread for each machine to be monitored and grabs each health metric. Results are printed to STDOUT so you can pipe it to a file (it is in csv format) to analyze. You can call this script at regular intervals to continuously monitor a set of machines.
#!/usr/bin/env python
# (c) Corey Goldberg (corey@goldb.org) 2008
import time
import threading
import pythoncom
import machine_stats
# set these to the Windows login name with admin privileges
user_name = 'corey'
pwd = 'secret'
def main():
host_names = [line.strip() for line in open('machines.txt', 'r').readlines()]
metric_store = MetricStore()
# gather metrics in threads
for host in host_names:
mp = MetricsProbe(host, metric_store)
mp.start()
while len(threading.enumerate()) > 1: # wait for threads to finish
time.sleep(.25)
class MetricsProbe(threading.Thread):
def __init__(self, host, metric_store):
threading.Thread.__init__(self)
self.host = host
self.metric_store = metric_store
def run(self):
pythoncom.CoInitialize() # need this for multithreading COM/WMI
try:
uptime = machine_stats.get_uptime(self.host, user_name, pwd)
cpu = machine_stats.get_cpu(self.host, user_name, pwd)
mem_mbyte = machine_stats.get_mem_mbytes(self.host, user_name, pwd)
mem_pct = machine_stats.get_mem_pct(self.host, user_name, pwd)
except:
uptime = 0
cpu = 0
mem_mbyte = 0
mem_pct = 0
print 'error getting stats from host: ' + self.host
self.metric_store.uptimes[self.host] = uptime
self.metric_store.cpus[self.host] = cpu
self.metric_store.mem_mbytes[self.host] = mem_mbyte
self.metric_store.mem_pcts[self.host] = mem_pct
print '%s,uptime:%d,cpu:%d,mem_mbyte:%d,mem_pct:%d' \
% (self.host, uptime, cpu, mem_mbyte, mem_pct)
class MetricStore(object):
def __init__(self):
self.uptimes = {}
self.cpus = {}
self.mem_mbytes = {}
self.mem_pcts = {}
if __name__ == '__main__':
main()
Output is a single line like this:
12/15/08 08:10:33,server2,uptime:158,cpu:5,mem_mbyte:1125,mem_pct:29
Usage Instructions:
You could continuously pipe the output to text file like this to build your results file:
$> python monitor_machines.py >> output.txt
Below is a simple Python module for remotely monitoring Windows machines. It is used to retrieve performance/health metrics.
I only implemented 5 functions:
To run this, you will first need to install the WMI module. Follow the instructions and get the download from here: http://timgolden.me.uk/python/wmi.html (hint: it is just a single script you can drop in the same directory your script runs from)
import re
import wmi
from subprocess import Popen, PIPE
def get_uptime(computer, user, password):
c = wmi.WMI(computer=computer, user=user, password=password, find_classes=False)
secs_up = int([uptime.SystemUpTime for uptime in c.Win32_PerfFormattedData_PerfOS_System()][0])
hours_up = secs_up / 3600
return hours_up
def get_cpu(computer, user, password):
c = wmi.WMI(computer=computer, user=user, password=password, find_classes=False)
utilizations = [cpu.LoadPercentage for cpu in c.Win32_Processor()]
utilization = int(sum(utilizations) / len(utilizations)) # avg all cores/processors
return utilization
def get_mem_mbytes(computer, user, password):
c = wmi.WMI(computer=computer, user=user, password=password, find_classes=False)
available_mbytes = int([mem.AvailableMBytes for mem in c.Win32_PerfFormattedData_PerfOS_Memory()][0])
return available_mbytes
def get_mem_pct(computer, user, password):
c = wmi.WMI(computer=computer, user=user, password=password, find_classes=False)
pct_in_use = int([mem.PercentCommittedBytesInUse for mem in c.Win32_PerfFormattedData_PerfOS_Memory()][0])
return pct_in_use
def ping(host_name):
p = Popen('ping -n 1 ' + host_name, stdout=PIPE)
m = re.search('Average = (.*)ms', p.stdout.read())
if m:
return True
else:
raise Exception
I am building some home-brewed monitoring tools that use these functions as a basis for plugins.
How are other people monitoring Windows from Python?
Suggestions?
Improvements?
Note to self:
Here is a really good collection of remote WMI monitoring scripts for Windows:
It is written in C# and looks nicely done.
About 6 years ago, I spent a good chunk of time as a webmaster for the GNU Project and the Free Software Foundation. Volunteering for the FSF and GNU was one way to express my ideals and somehow give back to the community. This is also what prompts me to release most of my own software under the GPL. The ethics and technical merits of Free Software have appealed to me since I was first introduced to the concept about a decade ago.
Here is the latest newsletter from the FSF regarding their current campaigns. It is a great summary of what is going on in the world of Free Software right now:
The Big Push 2009 -- Free Software Foundation Appeal
"As advocates for free software, we can challenge the status quo and so-called convenience of using the invasive tools of proprietary software companies, because the opportunities for change have never been better."
-Peter Brown
Executive Director, Free Software Foundation
I normally don't use an IDE for Python development. The majority of my day is spent inside my SciTE editor.
However, Sun recently released NetBeans IDE 6.5 for Python, and I figured I would give it a try. So far I really like it. The Python support works well. It has syntax highlighting, code complete, debugger, integrated console, etc. Overall, it is a comprehensive and nicely designed IDE.
NetBeans is a full featured development environment and supports:
Ajax | C/C++ | Databases | Debugger | Desktop | Editor | Groovy | GUI Builder | Java EE | JavaFX | Java ME | Java SE | JavaScript | Mobile | PHP | Profiler | Python | Refactor | REST | Rich Client Platform | Ruby | SOA | SOAP | UML | Web | WSDL | XML
The IDE is written in Java so I thought it would be dog slow, but it's pretty snappy to work with.
I've decided I will try it for a week and then decide whether to use it as my main dev environment.
Anyone else using NetBeans for Python development?
At my current gig, these are the essential tools I use for almost all development:
IDE/Editors:
Language Compilers/Interpreters:
Utilities/Etc:
I heard this term for the first time today, and realized it is a habit I usually follow rather than firing up a debugger and stepping through my code.
Defined by Dennis Lee Bieber here:
http://www.velocityreviews.com/forums/t356093-how-to-debug-python-code.html
"There's one wolf in Alaska, how do you find it? First build a fence down the middle of the state, wait for the wolf to howl, determine which side of the fence it is on. Repeat process on that side only, until you get to the point where you can see the wolf). In other words; put in a few "print" statements until you find the statement that is failing (then maybe work backwords from the "tracks" to find out where the wolf/bug comes from)."
The original term comes from "The Wolf Fence algorithm for debugging" (Edward J. Gauss, ACM - 1982)
I had previously tested Pylot on various machines and OS's, including: Windows XP, Vista, Cygwin, Ubuntu, Eee PC.
Recently I got news that it also runs on Macs, and the GUI looks pretty damn good:
This is a little script that gets the CPU Utilization metric from a remote Windows machine. On multi-way or multi-core machines, it will return an average of all processors/cores.
To run this, you will first need to install the WMI module. Follow the instructions and get the download from here: http://timgolden.me.uk/python/wmi.html
import wmi
def get_cpu(computer, user, password):
c = wmi.WMI(find_classes=False, computer=computer, user=user, password=password)
utilizations = [cpu.LoadPercentage for cpu in c.Win32_Processor()]
utilization = int(sum(utilizations) / len(utilizations))
return utilization
15 Tools to Help You Develop Faster Web Pages
"Response times, availability, and stability are vital factors to bear in mind when creating and maintaining a web application. If you’re concerned about your web pages’ speed or want to make sure you’re in tip-top shape before starting or launching a project, here’s a few useful, free tools to help you create and sustain high-performance web applications."
Pylot makes the list :)
Pylot (my web performance test tool) and WebInject (my web functional test tool) were both just named on the "List of top web application load and stress test tools" from WarePrise.
I am really glad people are finding Pylot useful. It still needs a lot of work and functionality to get it to where I want it to be, but so far adoption is good for such a simple tool. It is written in Python and I use it regularly for my own testing.
WebInject is a more advanced tool that has been widely adopted for a few years as a testing tool and monitoring plugin. Lately it has taken on a life of its own and keeps popping up everywhere. Who knew my ugly ball of Perl code would get that popular?
Thanks for the props!
Paul Graham has some great essays online that made up the content of his book: Hackers & Painters. One of my favorite posts is The Python Paradox, which presents something rather counterintuitive at first read:
"I'll call it the Python paradox: if a company chooses to write its software in a comparatively esoteric language, they'll be able to hire better programmers, because they'll attract only those who cared enough to learn it. And for programmers the paradox is even more pronounced: the language to learn, if you want to get a good job, is a language that people don't learn merely to get a job.
This concept may be a little scary to some. Learning an esoteric language improves your chances at getting a "good" job? (presumably "good" meaning one you like) But what about all the recruiters salivating for Java/JEE and .NET programmers?
"People don't learn Python because it will get them a job; they learn it because they genuinely like to program and aren't satisfied with the languages they already know."
.. and there lies the paradox. If you don't know an esoteric language, or one that is not considered mainstream Enterprise, you can't learn one just to land a job. You must already happen to enjoy programming enough to seek it on your own.
I just made my 3000th comment/post over at SQAForums. I've been a member and have enjoyed being part of that community since 2001. 8 years and 3000 posts.. wow.
I spend a lot of time posting in the "Performance & Load Testing" and "Automated Testing" forums. Good stuff.
"This article shows how response times improved after upgrading to a new dedicated web server with solid-state drives and faster processors."http://www.websiteoptimization.com/speed/tweak/solid-state-drive/
This state machine diagram from the Python source code (client.py in Python 3, httplib.py in Python 2) is useful for understanding the different states of a client/server connection via HTTP:
http://svn.python.org/view/python/branches/py3k/Lib/http/
HTTPConnection goes through a number of "states", which define when a client
may legally make another request or fetch the response for a particular
request. This diagram details these state transitions:
(null)
|
| HTTPConnection()
v
Idle
|
| putrequest()
v
Request-started
|
| ( putheader() )* endheaders()
v
Request-sent
|
| response = getresponse()
v
Unread-response [Response-headers-read]
|\____________________
| |
| response.read() | putrequest()
v v
Idle Req-started-unread-response
______/|
/ |
response.read() | | ( putheader() )* endheaders()
v v
Request-started Req-sent-unread-response
|
| response.read()
v
Request-sent
This diagram presents the following rules:
-- a second request may not be started until {response-headers-read}
-- a response [object] cannot be retrieved until {request-sent}
-- there is no differentiation between an unread response body and a
partially read response body
Python 2.6 is official:
http://www.python.org/download/releases/2.6/
What's new in Python 2.6:
http://docs.python.org/whatsnew/2.6.html
Thanks to all the devs and contributors.. you make my programming life great.
A good writeup here:
http://jessenoller.com/2008/10/02/python-26-is-released/
I have lots of references and books for Python 2.x. I am looking forward to upgrading them and doing lots of reading for the upcoming Python 3.0 release.
As usual, the official Python 3.0 docs are great:
http://docs.python.org/dev/3.0/
There is already a Python 3.0 book online by Swaroop C H:
A Byte Of Python
I also just pre-ordered Mark Summerfield's printed book on Python 3.0:
Programming in Python 3: A Complete Introduction to the Python Language
Hopefully lots more books and docs on Python 3.0 are coming soon.
I've been having fun using the new StackOverflow site for answering technical questions.
Here is a Python script I call 'StackOverlow Fight' (like Google Fight). It takes a list of tags and gets the count of questions that are tagged with each one.
As an example, here is StackOverflow fight between some popular programming languages:
#!/usr/bin/env python
import urllib
import re
langs = ('python', 'ruby', 'perl', 'java')
url_stem = 'http://stackoverflow.com/questions/tagged/'
counts = {}
for lang in langs:
resp = urllib.urlopen(url_stem + lang).read()
m = re.search('summarycount.*>(.*)<', resp)
count = int(m.group(1).replace(',', ''))
counts[lang] = count
print lang, ':', count
sorted_counts = sorted(counts.items(), key=lambda(k,v):(v,k))
sorted_counts.reverse()
print sorted_counts[0][0], 'wins with', sorted_counts[0][1]
Output:
python : 733 ruby : 391 perl : 167 java : 1440 java wins with 1440
This is the best article on thread synchronization in Python that I have come across: Thread Synchronization Mechanisms in Python
It is very clearly explained and just helped me solve a major concurrency issues I was having.
Especially interesting is the list of thread safe operations in Python.
Here are some thread-safe operations:
- reading or replacing a single instance attribute
- reading or replacing a single global variable
- fetching an item from a list
- modifying a list in place (e.g. adding an item using append)
- fetching an item from a dictionary
- modifying a dictionary in place (e.g. adding an item, or calling the clear method)
http://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm
Update: Now I am confused. Doesn't this example in the article contradict bullet point 2 above?
Update Update: See comments below for better explanation and info.
Stack Overflow is a new community site for answering programming questions. It is very impressive to me, but there seems to be a very Microsoft (.NET, C#, etc) slant to the questions being asked.
It would be great to see more Python questions and answers over there. The one python related question I did ask was answered fabulously... so there are Python-heads reading and answering, but not posting very many questions.
Fellow Pythonistas, hop to it! Let's get a bigger Python presence over there.
I have been saying this for years, and Alan Page finally posted about it:
GUI Schmooey"The point is that I think testers, in general, spend too much time trying to automate GUIs. I’m not against automation – just write automation starting at a level below the GUI and move down from there."
The GUI tends to be the most fragile layer of an application. It also tends to be the layer where a lot of automated test infrastructure is built. I don't like this style of "black box automation". It is tempting to automate a regular user's UI experience; and indeed this can be useful. However, in most cases, you are not being as productive as you could be by taking a manual/unscripted approach to your GUI functional testing.
Manual and Exploratory testing seems like a good fit for GUI testing, while lower layers can benefit most from test automation.
Dear Lazyweb,
Which is better to use for timing in Python? time.clock() or time.time()? Which one provides more accuracy?
for example:
start = time.clock() ... do something elapsed = (time.clock() - start)
vs.
start = time.time() ... do something elapsed = (time.time() - start)
Update:
From the comments I received and others I have talked with, it looks like the answer is: use time() on Unix/Linux systems and clock() on Windows systems. However, the 'timeit' module gives you more options and is accurate across platforms.
Wow.. talk about programming with a ball and chain on! Some languages just take so much code to do a simple thing.
The following 2 programs each do an HTTP GET and print the contents of the URL.
In Java:
import java.net.*;
import java.io.*;
public class JGet {
public static void main (String[] args) throws IOException {
try {
URL url = new URL("http://www.google.com");
BufferedReader in =
new BufferedReader(new InputStreamReader(url.openStream()));
String str;
while ((str = in.readLine()) != null) {
System.out.println(str);
}
in.close();
}
catch (MalformedURLException e) {}
catch (IOException e) {}
}
}
Equivalent in Python:
import urllib
print urllib.urlopen('http://www.google.com').read()
I've been programming almost exclusively in Python for the past 4 years or so. Occasionally I have to write code in C# or Perl.
After using Python:
I love me some Python... but dammit... you ruined me.
Here is a story and some observations I have made as an Open Source developer:
I had a specific need for a tool to test web services a few years ago ('02-'03). I had been rolling my own tools in various languages for quite a while and had some stuff that I thought others might be interested in; so I setup a SourceForge project and released it in Jan'04. The tool is called WebInject. Not many people used it at first. I just kept adding features and releasing. I was scratching my own itches and using my tool internally at a company I was working for. Eventually I got a few bites and some people started to download it and try it out. As of today, the tool has been downloaded over 60,000 times. I've had feedback from all corners of the world.
In the early days, *any* bug that was reported, I would dive into immediately. I was turning around fixes and new releases in crazy time. This personal attention sparked a lot of lengthy email conversations where I was starting to get real feedback. It wasn't the case where I could just sit back and expect quality feedback. I really engaged the users and asked questions. Pretty soon I was getting more feedback than I could reasonably handle. Around this time I started getting some patches sent to me. This accelerated development as I started to get various ideas from other people. A few silly bugs were also cleaned up; stuff I never would have caught on my own. Encouraging patches is huge. Once someone gets a piece of their own code into your code base, they have an intimate connection.
I don't maintain WebInject much any more. I think it is cool and useful, but it's a rats nest of Perl code that needs some serious love. There is some stuff that bothers me so much about it that I barely want to touch it sometimes :) I spend all my time on newer tools these days, but I still keep on top of it enough to facilitate others using it and posting patches/updates to it (though I haven't released in a long time). Oddly, it has become somewhat entrenched in monitoring systems. Most of the users lately seem to be people running Nagios (open source monitoring system) that need an intelligent web plugin/agent.
Some basic thoughts on open source tool adoption and getting feedback:
My web testing tool (WebInject) is now being used as part of the Advanced Java class being taught at SMU's Engineering school.
They will be using it for testing Servlets.
very cool!
http://engr.smu.edu/~coyle/cse7345/This is a an ordered list of programming languages I have learned in the past 15 years:
#!/usr/bin/env python
# ssl_load.py - Corey Goldberg - 2008
import httplib
from threading import Thread
threads = 250
host = '192.168.1.14'
file = '/foo.html'
def main():
for i in range(threads):
agent = Agent()
agent.start()
class Agent(Thread):
def __init__(self):
Thread.__init__(self)
def run(self):
while True:
conn = httplib.HTTPSConnection(host)
conn.request('GET', file)
resp = conn.getresponse()
if __name__ == '__main__':
main()
This works great for small workloads, but Windows stops me from creating threads at around 800 per process (sometimes it was flaky and crashed after 300), so that won't work. The next step was to break the load into several processes and launch threads from each process. Rather than try to wrap this into a single script, I just use a starter script to call many instances of my load test script. I used a starter script like this:#!/usr/bin/env python
import subprocess
processes = 60
for i in range(processes):
subprocess.Popen('python ssl_load.py')
250 threads per process seemed about right and didn't crash my system. Using this, I was able to launch 60 processes which gave me a total of 15k threads running."If I pay to connect to the Net with a given quality of service, and you pay to connect to the Net with the same or higher quality of service; Then you and I can communicate across the net with that quality of service. That's all."
"Freedom of connection with any application to any party."
import urllib
url = 'http://www.goldb.org/images/goldb.png'
f = urllib.urlopen(url)
fh = open('goldb.png', 'wb')
fh.write(f.read())
fh.close()
"All Linux distributions ship with: Python, Perl, C, and C++"
"A standard install of Mac OS X Leopard has: Python, Perl, Java, Ruby, AppleScript"
"MS Windows ships with nothing. No BASIC. No C compiler. You're trapped, stuck playing with Solitaire and MS Paint"
foo = [1, 2, 1, 2, 1, 2, 1, 1]... and you want to count the items in the list, ending up with a dictionary full of items (dict keys) and counts (dict values), like this:
{1: 5, 2: 3}
Here is an easy way to do it:
d = {}
for i in set(foo):
d[i] = foo.count(i)
got anything more Pythonic?
{'a': 2,
'b': 4,
'c': 3,
'd': 1}
.. and you want to convert it into a list, sorted by value (not key), like this:
[('d', 1),
('a', 2),
('c', 3),
('b', 4)]
There is a Python Recipe for it with various implementations discussed in the comments.sorted(foo.items(), key=lambda(k,v):(v,k))
SELECT count(*) FROM :: fn_trace_getinfo(default) WHERE property = 5 and value = 1Here is how you can find more detail about the running traces:
SELECT * FROM :: fn_trace_getinfo(default)You can terminate a trace with the 'sp_trace_setstatus' stored procedure using the traceid:
EXEC sp_trace_setstatus 1, @status = 0 EXEC sp_trace_setstatus 1, @status = 2setting the status to 0 stops the trace setting the status to 2 closes the trace and deletes its definition from the server
"Pylot is an open-source performance and scalability testing tool. It uses HTTP load tests so that you can plan, benchmark, analyze and tweak performance. Pylot requires that you have Python installed on the server - but you don’t need to know the language, you use XML to create your testing scenarios."
import ystockquote
ticker = 'GOOG'
start = '20080101'
end = '20080523'
data = ystockquote.get_historical_prices(ticker, start, end)
closing_prices = [x[4] for x in data][1:]
closing_prices.reverse()
fh = open('data.txt', 'w')
for closing_price in closing_prices:
fh.write(closing_price + '\n')
fh.close()
Now that you have the data.txt file setup, you can call the Sparkplot script via the command line to generate the sparkline image:
python sparkplot.py --label_first --label_last*Note: The Sparkplot module needs Matplotlib installed. The sparkline output is an image (png) like this:
import java.io.*;
public static String cmdExec(String cmdLine) {
String line;
String output = "";
try {
Process p = Runtime.getRuntime().exec(cmdLine);
BufferedReader input = new BufferedReader
(new InputStreamReader(p.getInputStream()));
while ((line = input.readLine()) != null) {
output += (line + '\n');
}
input.close();
}
catch (Exception ex) {
ex.printStackTrace();
}
return output;
}
Sample Usage:
public static void main(String[] args) {
CmdExec cmd = new CmdExec();
System.out.println(cmd.run("ls -a"));
}
"RRDtool is the Open Source industry standard, high performance data logging and graphing system for time series data. It stores the data in a very compact way that will not expand over time, and it can create beautiful graphs."If you are developing tools that need a data repository and graphing capabilities, RRDTool provides you both. You create an RRD and then you insert data values at regular intervals. You then call the graphing API to have a graph displayed. The cool thing about this data storage is its “round robin” nature. You define various time spans and the granularity at which you want them stored. A fixed binary file is created, and this never grows in size over time. As you insert more data, it is inserted into each span. As results are collected, they are averaged and rolled into successive time spans. It makes a much more efficient system than using your own complex data structures, relational databases, or file system storage.
import rrd interval = 10 rrd_file = 'test.rrd' my_rrd = rrd.RRD(rrd_file, vertical_label='value') my_rrd.create_rrd(interval)Now that I have my RRD created, what can I do with it? For a quick example, I can use the rrd_feeder_rand.py script to feed in random numbers and generate a graph every 10 seconds. The graph generated will show the past 60 mins of data. The code for that would look like:
import rrd
import random
import time
interval = 10
rrd_file = 'test.rrd'
my_rrd = rrd.RRD(rrd_file)
while True:
rand = random.randint(1, 100)
my_rrd.update(rand)
my_rrd.graph(60)
time.sleep(interval)
OK, pretty boring. How about an HTTP website monitor?
I created the rrd_feeder_http.py script to show how this is done. This script will send HTTP GET requests to a specified url. A request is sent every 10 seconds and a graph of response times for the past hour is generated.
import rrd
import time
import httplib
host = 'www.python.org'
path = '/'
use_ssl = False
interval = 10
rrd_file = 'test.rrd'
def main():
my_rrd = rrd.RRD(rrd_file, 'Response Time')
while True:
start_time = time.time()
if send(host):
end_time = time.time()
raw_latency = end_time - start_time
expire_time = (interval - raw_latency)
latency = ('%.3f' % raw_latency)
my_rrd.update(latency)
my_rrd.graph(60)
print latency
else:
expire_time = interval
if expire_time > 0:
time.sleep(expire_time)
def send(host):
if use_ssl:
conn = httplib.HTTPSConnection(host)
else:
conn = httplib.HTTPConnection(host)
try:
conn.request('GET', path)
body = conn.getresponse().read()
return True
except:
print 'Failed request'
return False
if __name__ == '__main__':
main()
The output from running the HTTP website monitor script looks like this:
import ystockquote
ticker = 'GOOG'
start = '20080520'
end = '20080523'
data = ystockquote.get_historical_prices(ticker, start, end)
for dat in data:
print dat
Output:
['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Clos'] ['2008-05-23', '546.96', '553.00', '537.81', '544.62', '4431500', '544.6'] ['2008-05-22', '551.95', '554.21', '540.25', '549.46', '5076300', '549.4'] ['2008-05-21', '578.52', '581.41', '547.89', '549.99', '6468100', '549.9'] ['2008-05-20', '574.63', '582.48', '572.91', '578.60', '3313600', '578.6']