Over at "Inside the Nerdery" (love the name), Jodi Chromey did a short writeup on web performance and analysis tools. Pylot made the list :)
6 tools you can use to analyze your Web site’s performance
"Everyone wants a Web site that’s running at optimum speed, but for a lot of people they have no idea what the heck that is. That’s why the Nerds have created this list of six Web performance analysis tools."
I've been writing code in Python 3.0 (3000) this week. It feels really nice. Basically the same old Python, with some corners cleaned up and reorganized.
I have tons of legacy code to support that is written in 2.4/2.5, so I still need to keep my hands in Python 2.x for quite a long time. But when will I switch over all new projects so they use Python 3.0?
Well the answer is: as soon as my 5 favorite libraries work with Python 3.0. Here is what I'm waiting on:
I have no idea when these will be ported to work with 3.0. Hopefully I'm not waiting *years* for this.
When are you switching or migrating?
My monitoring tool: WebInject is a popular plugin in the Nagios community. A Nagios user (Felipe Ferreira) just wrote a small how-to on setting up WebInject. (though the official manual is much more thorough)
Webinject How to
"When monitoring websites, many times we will need more than just checking if the site is up. We may need to see if the internals of the website are working. A good example is making a user login check. Using Nagios alone it can not be done, but thanks to Corey Goldberg, it is possible using his script webinject.pl. [snip...] And yes it works with HTTPS and redirections."
Thanks Felipe!
Here is a Python script to reboot a remote Windows server. It requires the Python Win32 Extensions to be installed.
# Reboot a remove server
import win32security
import win32api
import sys
import time
from ntsecuritycon import *
def RebootServer(message='Server Rebooting', timeout=30, bForce=0, bReboot=1):
AdjustPrivilege(SE_SHUTDOWN_NAME)
try:
win32api.InitiateSystemShutdown(None, message, timeout, bForce, bReboot)
finally:
# Remove the privilege we just added.
AdjustPrivilege(SE_SHUTDOWN_NAME, 0)
def AbortReboot():
AdjustPrivilege(SE_SHUTDOWN_NAME)
try:
win32api.AbortSystemShutdown(None)
finally:
# Remove the privilege we just added.
AdjustPrivilege(SE_SHUTDOWN_NAME, 0)
def AdjustPrivilege(priv, enable=1):
# Get the process token.
flags = TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY
htoken = win32security.OpenProcessToken(win32api.GetCurrentProcess(), flags)
# Get the ID for the system shutdown privilege.
id = win32security.LookupPrivilegeValue(None, priv)
# Obtain the privilege for this process.
# Create a list of the privileges to be added.
if enable:
newPrivileges = [(id, SE_PRIVILEGE_ENABLED)]
else:
newPrivileges = [(id, 0)]
# and make the adjustment.
win32security.AdjustTokenPrivileges(htoken, 0, newPrivileges)
if __name__=='__main__':
message = 'This server is pretending to reboot\nThe shutdown will stop in 10 seconds'
RebootServer(message)
print 'Sleeping for 10 seconds'
time.sleep(10)
print 'Aborting shutdown'
AbortReboot()
Note: This code came from somewhere on the net and was not attributed to anyone. If you wrote it, let me know.
Holiday reading just arrived...
Finally got my hands on the brand new "Programming in Python 3 - A Complete Introduction to the Python Language". As far as I know, this is the first print book covering Python 3.0 (Python 3000). A quick skim looked promising.
Update: I've read a good chunk of this book now, and it is excellent. It's very clearly written and all code examples are clean and follow nice conventions. It covers a wide variety of topics including all the important stuff in the language and standard library. This book would make a good mid level introductory book to the language, or as a good reference/refresher to a more experience Python hacker. I highly recommend picking this up.
I've finally realized...
REST is a godsend for Performance and Load Testing. It's not just a steaming pile of crap tunneled over HTTP.
Dropbox is a file synching and backup tool that I began using a few months back. It is now one of my favorite tools.
You install a service and designate a folder as your "dropbox" and then register the machine with your online account. You can do this on as many machines as you want. When you add content to the folder, all the registered machines with it installed synch the folder together.
So I have one common "code" folder on my 2 home laptops and work PC that I keep in synch. It also stores the data in a secure web repository and has version control.
Dropbox is useful for synching multiple machines or collaborating with a group of people that have synched dropboxes to the same account. It works over the web on port 80 so you don't have to worry about VPN's or corporate firewalls or anything. It also works on almost all platforms and is free (up to 2GB storage).
Wordle generates "word clouds" from text that you provide. Here is a Wordle mashup showing the word density in my Tweets from Twitter:
In a recent blog post, I introduced a script for remotely monitoring Windows machines. It is used to retrieve performance/health metrics. It uses WMI (Windows Management Instrumentation) to interact with the Windows machine and its Win32 classes.
Now I will show how you an example of using the module (which I named "machine_stats.py"). I call it from a multithreaded controller to poll multiple machines in parallel.
The script below reads a list of Host Names (or IP's) from a file named "machines.txt". It then spawns a thread for each machine to be monitored and grabs each health metric. Results are printed to STDOUT so you can pipe it to a file (it is in csv format) to analyze. You can call this script at regular intervals to continuously monitor a set of machines.
#!/usr/bin/env python
# (c) Corey Goldberg (corey@goldb.org) 2008
import time
import threading
import pythoncom
import machine_stats
# set these to the Windows login name with admin privileges
user_name = 'corey'
pwd = 'secret'
def main():
host_names = [line.strip() for line in open('machines.txt', 'r').readlines()]
metric_store = MetricStore()
# gather metrics in threads
for host in host_names:
mp = MetricsProbe(host, metric_store)
mp.start()
while len(threading.enumerate()) > 1: # wait for threads to finish
time.sleep(.25)
class MetricsProbe(threading.Thread):
def __init__(self, host, metric_store):
threading.Thread.__init__(self)
self.host = host
self.metric_store = metric_store
def run(self):
pythoncom.CoInitialize() # need this for multithreading COM/WMI
try:
uptime = machine_stats.get_uptime(self.host, user_name, pwd)
cpu = machine_stats.get_cpu(self.host, user_name, pwd)
mem_mbyte = machine_stats.get_mem_mbytes(self.host, user_name, pwd)
mem_pct = machine_stats.get_mem_pct(self.host, user_name, pwd)
except:
uptime = 0
cpu = 0
mem_mbyte = 0
mem_pct = 0
print 'error getting stats from host: ' + self.host
self.metric_store.uptimes[self.host] = uptime
self.metric_store.cpus[self.host] = cpu
self.metric_store.mem_mbytes[self.host] = mem_mbyte
self.metric_store.mem_pcts[self.host] = mem_pct
print '%s,uptime:%d,cpu:%d,mem_mbyte:%d,mem_pct:%d' \
% (self.host, uptime, cpu, mem_mbyte, mem_pct)
class MetricStore(object):
def __init__(self):
self.uptimes = {}
self.cpus = {}
self.mem_mbytes = {}
self.mem_pcts = {}
if __name__ == '__main__':
main()
Output is a single line like this:
12/15/08 08:10:33,server2,uptime:158,cpu:5,mem_mbyte:1125,mem_pct:29
Usage Instructions:
You could continuously pipe the output to text file like this to build your results file:
$> python monitor_machines.py >> output.txt
Below is a simple Python module for remotely monitoring Windows machines. It is used to retrieve performance/health metrics.
I only implemented 5 functions:
To run this, you will first need to install the WMI module. Follow the instructions and get the download from here: http://timgolden.me.uk/python/wmi.html (hint: it is just a single script you can drop in the same directory your script runs from)
import re
import wmi
from subprocess import Popen, PIPE
def get_uptime(computer, user, password):
c = wmi.WMI(computer=computer, user=user, password=password, find_classes=False)
secs_up = int([uptime.SystemUpTime for uptime in c.Win32_PerfFormattedData_PerfOS_System()][0])
hours_up = secs_up / 3600
return hours_up
def get_cpu(computer, user, password):
c = wmi.WMI(computer=computer, user=user, password=password, find_classes=False)
utilizations = [cpu.LoadPercentage for cpu in c.Win32_Processor()]
utilization = int(sum(utilizations) / len(utilizations)) # avg all cores/processors
return utilization
def get_mem_mbytes(computer, user, password):
c = wmi.WMI(computer=computer, user=user, password=password, find_classes=False)
available_mbytes = int([mem.AvailableMBytes for mem in c.Win32_PerfFormattedData_PerfOS_Memory()][0])
return available_mbytes
def get_mem_pct(computer, user, password):
c = wmi.WMI(computer=computer, user=user, password=password, find_classes=False)
pct_in_use = int([mem.PercentCommittedBytesInUse for mem in c.Win32_PerfFormattedData_PerfOS_Memory()][0])
return pct_in_use
def ping(host_name):
p = Popen('ping -n 1 ' + host_name, stdout=PIPE)
m = re.search('Average = (.*)ms', p.stdout.read())
if m:
return True
else:
raise Exception
I am building some home-brewed monitoring tools that use these functions as a basis for plugins.
How are other people monitoring Windows from Python?
Suggestions?
Improvements?
Note to self:
Here is a really good collection of remote WMI monitoring scripts for Windows:
It is written in C# and looks nicely done.
About 6 years ago, I spent a good chunk of time as a webmaster for the GNU Project and the Free Software Foundation. Volunteering for the FSF and GNU was one way to express my ideals and somehow give back to the community. This is also what prompts me to release most of my own software under the GPL. The ethics and technical merits of Free Software have appealed to me since I was first introduced to the concept about a decade ago.
Here is the latest newsletter from the FSF regarding their current campaigns. It is a great summary of what is going on in the world of Free Software right now:
The Big Push 2009 -- Free Software Foundation Appeal
"As advocates for free software, we can challenge the status quo and so-called convenience of using the invasive tools of proprietary software companies, because the opportunities for change have never been better."
-Peter Brown
Executive Director, Free Software Foundation
I normally don't use an IDE for Python development. The majority of my day is spent inside my SciTE editor.
However, Sun recently released NetBeans IDE 6.5 for Python, and I figured I would give it a try. So far I really like it. The Python support works well. It has syntax highlighting, code complete, debugger, integrated console, etc. Overall, it is a comprehensive and nicely designed IDE.
NetBeans is a full featured development environment and supports:
Ajax | C/C++ | Databases | Debugger | Desktop | Editor | Groovy | GUI Builder | Java EE | JavaFX | Java ME | Java SE | JavaScript | Mobile | PHP | Profiler | Python | Refactor | REST | Rich Client Platform | Ruby | SOA | SOAP | UML | Web | WSDL | XML
The IDE is written in Java so I thought it would be dog slow, but it's pretty snappy to work with.
I've decided I will try it for a week and then decide whether to use it as my main dev environment.
Anyone else using NetBeans for Python development?
At my current gig, these are the essential tools I use for almost all development:
IDE/Editors:
Language Compilers/Interpreters:
Utilities/Etc:
I heard this term for the first time today, and realized it is a habit I usually follow rather than firing up a debugger and stepping through my code.
Defined by Dennis Lee Bieber here:
http://www.velocityreviews.com/forums/t356093-how-to-debug-python-code.html
"There's one wolf in Alaska, how do you find it? First build a fence down the middle of the state, wait for the wolf to howl, determine which side of the fence it is on. Repeat process on that side only, until you get to the point where you can see the wolf). In other words; put in a few "print" statements until you find the statement that is failing (then maybe work backwords from the "tracks" to find out where the wolf/bug comes from)."
The original term comes from "The Wolf Fence algorithm for debugging" (Edward J. Gauss, ACM - 1982)
