December 30, 2008

Pylot - 6 Tools You Can Use To Analyze Your Web Site's Performance

Over at "Inside the Nerdery" (love the name), Jodi Chromey did a short writeup on web performance and analysis tools. Pylot made the list :)

6 tools you can use to analyze your Web site’s performance

"Everyone wants a Web site that’s running at optimum speed, but for a lot of people they have no idea what the heck that is. That’s why the Nerds have created this list of six Web performance analysis tools."

December 29, 2008

When Will I Officially Switch To Python 3.0?

I've been writing code in Python 3.0 (3000) this week. It feels really nice. Basically the same old Python, with some corners cleaned up and reorganized.

I have tons of legacy code to support that is written in 2.4/2.5, so I still need to keep my hands in Python 2.x for quite a long time. But when will I switch over all new projects so they use Python 3.0?

Well the answer is: as soon as my 5 favorite libraries work with Python 3.0. Here is what I'm waiting on:

I have no idea when these will be ported to work with 3.0. Hopefully I'm not waiting *years* for this.

When are you switching or migrating?

December 28, 2008

WebInject - Web Monitoring - Getting It Working

My monitoring tool: WebInject is a popular plugin in the Nagios community. A Nagios user (Felipe Ferreira) just wrote a small how-to on setting up WebInject. (though the official manual is much more thorough)

Webinject How to

"When monitoring websites, many times we will need more than just checking if the site is up. We may need to see if the internals of the website are working. A good example is making a user login check. Using Nagios alone it can not be done, but thanks to Corey Goldberg, it is possible using his script webinject.pl. [snip...] And yes it works with HTTPS and redirections."

Thanks Felipe!

December 26, 2008

Python - Windows - Reboot a Remote Server

Here is a Python script to reboot a remote Windows server. It requires the Python Win32 Extensions to be installed.

# Reboot a remove server

import win32security
import win32api
import sys
import time
from ntsecuritycon import *


def RebootServer(message='Server Rebooting', timeout=30, bForce=0, bReboot=1):
    AdjustPrivilege(SE_SHUTDOWN_NAME)
    try:
        win32api.InitiateSystemShutdown(None, message, timeout, bForce, bReboot)
    finally:
        # Remove the privilege we just added.
        AdjustPrivilege(SE_SHUTDOWN_NAME, 0)


def AbortReboot():
    AdjustPrivilege(SE_SHUTDOWN_NAME)
    try:
        win32api.AbortSystemShutdown(None)
    finally:
        # Remove the privilege we just added.
        AdjustPrivilege(SE_SHUTDOWN_NAME, 0)


def AdjustPrivilege(priv, enable=1):
    # Get the process token.
    flags = TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY
    htoken = win32security.OpenProcessToken(win32api.GetCurrentProcess(), flags)
    # Get the ID for the system shutdown privilege.
    id = win32security.LookupPrivilegeValue(None, priv)
    # Obtain the privilege for this process.
    # Create a list of the privileges to be added.
    if enable:
        newPrivileges = [(id, SE_PRIVILEGE_ENABLED)]
    else:
        newPrivileges = [(id, 0)]
    # and make the adjustment.
    win32security.AdjustTokenPrivileges(htoken, 0, newPrivileges)
    
    
if __name__=='__main__':
        message = 'This server is pretending to reboot\nThe shutdown will stop in 10 seconds'
        RebootServer(message)
        print 'Sleeping for 10 seconds'
        time.sleep(10)
        print 'Aborting shutdown'
        AbortReboot()

Note: This code came from somewhere on the net and was not attributed to anyone. If you wrote it, let me know.

December 25, 2008

New Book - Programming in Python 3

Holiday reading just arrived...

Finally got my hands on the brand new "Programming in Python 3 - A Complete Introduction to the Python Language". As far as I know, this is the first print book covering Python 3.0 (Python 3000). A quick skim looked promising.

Update: I've read a good chunk of this book now, and it is excellent. It's very clearly written and all code examples are clean and follow nice conventions. It covers a wide variety of topics including all the important stuff in the language and standard library. This book would make a good mid level introductory book to the language, or as a good reference/refresher to a more experience Python hacker. I highly recommend picking this up.

December 24, 2008

REST Simplifies Performance and Load Testing

I've finally realized...

REST is a godsend for Performance and Load Testing. It's not just a steaming pile of crap tunneled over HTTP.

December 17, 2008

Dropbox - Great File Synching and Backup

Dropbox is a file synching and backup tool that I began using a few months back. It is now one of my favorite tools.

You install a service and designate a folder as your "dropbox" and then register the machine with your online account. You can do this on as many machines as you want. When you add content to the folder, all the registered machines with it installed synch the folder together.

So I have one common "code" folder on my 2 home laptops and work PC that I keep in synch. It also stores the data in a secure web repository and has version control.

Dropbox is useful for synching multiple machines or collaborating with a group of people that have synched dropboxes to the same account. It works over the web on port 80 so you don't have to worry about VPN's or corporate firewalls or anything. It also works on almost all platforms and is free (up to 2GB storage).

December 16, 2008

Wordle - Word Cloud Of My Twitter Tweets

Wordle generates "word clouds" from text that you provide. Here is a Wordle mashup showing the word density in my Tweets from Twitter:

December 15, 2008

Python - Get Remote Windows Metrics Script

In a recent blog post, I introduced a script for remotely monitoring Windows machines. It is used to retrieve performance/health metrics. It uses WMI (Windows Management Instrumentation) to interact with the Windows machine and its Win32 classes.

Now I will show how you an example of using the module (which I named "machine_stats.py"). I call it from a multithreaded controller to poll multiple machines in parallel.

The script below reads a list of Host Names (or IP's) from a file named "machines.txt". It then spawns a thread for each machine to be monitored and grabs each health metric. Results are printed to STDOUT so you can pipe it to a file (it is in csv format) to analyze. You can call this script at regular intervals to continuously monitor a set of machines.

#!/usr/bin/env python
# (c) Corey Goldberg (corey@goldb.org) 2008


import time
import threading
import pythoncom
import machine_stats



# set these to the Windows login name with admin privileges
user_name = 'corey'
pwd = 'secret'
        
        
        
def main():           
    host_names = [line.strip() for line in open('machines.txt', 'r').readlines()]
    metric_store = MetricStore()
               
    # gather metrics in threads
    for host in host_names:
        mp = MetricsProbe(host, metric_store)
        mp.start()
    while len(threading.enumerate()) > 1:  # wait for threads to finish
        time.sleep(.25)
    
        

class MetricsProbe(threading.Thread):
    def __init__(self, host, metric_store):
        threading.Thread.__init__(self)
        self.host = host
        self.metric_store = metric_store
        
    def run(self):
        pythoncom.CoInitialize()  # need this for multithreading COM/WMI
        try:
            uptime = machine_stats.get_uptime(self.host, user_name, pwd)
            cpu = machine_stats.get_cpu(self.host, user_name, pwd)
            mem_mbyte = machine_stats.get_mem_mbytes(self.host, user_name, pwd)
            mem_pct = machine_stats.get_mem_pct(self.host, user_name, pwd)            
        except:
            uptime = 0
            cpu = 0
            mem_mbyte = 0
            mem_pct = 0
            print 'error getting stats from host: ' + self.host
        self.metric_store.uptimes[self.host] = uptime
        self.metric_store.cpus[self.host] = cpu
        self.metric_store.mem_mbytes[self.host] = mem_mbyte
        self.metric_store.mem_pcts[self.host] = mem_pct
        print '%s,uptime:%d,cpu:%d,mem_mbyte:%d,mem_pct:%d' \
            % (self.host, uptime, cpu, mem_mbyte, mem_pct)
                


class MetricStore(object):
    def __init__(self):
        self.uptimes = {}
        self.cpus = {}
        self.mem_mbytes = {}
        self.mem_pcts = {}
            


if __name__ == '__main__':
    main()

Output is a single line like this:

12/15/08 08:10:33,server2,uptime:158,cpu:5,mem_mbyte:1125,mem_pct:29

Usage Instructions:

  • Download the Code from here: monitor_win_machines.zip
  • Edit "machines.txt" and add your own Hosts/IPs
  • Edit the username/password in "monitor_machines.py"
  • Run the "monitor_machines.py" script from the command line.

You could continuously pipe the output to text file like this to build your results file:

$> python monitor_machines.py >> output.txt

December 12, 2008

Python - Monitor Windows Remotely With WMI

Below is a simple Python module for remotely monitoring Windows machines. It is used to retrieve performance/health metrics.

I only implemented 5 functions:

  • Uptime
  • CPU Utilization
  • Available Memory
  • Memory Used
  • Ping
You can get an idea of how to interact with Windows via WMI (Windows Management Instrumentation). I use it from a multithreaded manager so I can concurrently poll a distributed farm of servers.

To run this, you will first need to install the WMI module. Follow the instructions and get the download from here: http://timgolden.me.uk/python/wmi.html (hint: it is just a single script you can drop in the same directory your script runs from)

import re
import wmi
from subprocess import Popen, PIPE



def get_uptime(computer, user, password):
    c = wmi.WMI(computer=computer, user=user, password=password, find_classes=False)
    secs_up = int([uptime.SystemUpTime for uptime in c.Win32_PerfFormattedData_PerfOS_System()][0])
    hours_up = secs_up / 3600
    return hours_up


def get_cpu(computer, user, password):
    c = wmi.WMI(computer=computer, user=user, password=password, find_classes=False)
    utilizations = [cpu.LoadPercentage for cpu in c.Win32_Processor()]
    utilization = int(sum(utilizations) / len(utilizations))  # avg all cores/processors
    return utilization

    
def get_mem_mbytes(computer, user, password):
    c = wmi.WMI(computer=computer, user=user, password=password, find_classes=False)
    available_mbytes = int([mem.AvailableMBytes for mem in c.Win32_PerfFormattedData_PerfOS_Memory()][0])
    return available_mbytes


def get_mem_pct(computer, user, password):
    c = wmi.WMI(computer=computer, user=user, password=password, find_classes=False)
    pct_in_use = int([mem.PercentCommittedBytesInUse for mem in c.Win32_PerfFormattedData_PerfOS_Memory()][0])
    return pct_in_use
    
    
def ping(host_name):
    p = Popen('ping -n 1 ' + host_name, stdout=PIPE)
    m = re.search('Average = (.*)ms', p.stdout.read())
    if m:
        return True
    else:
        raise Exception  

I am building some home-brewed monitoring tools that use these functions as a basis for plugins.

How are other people monitoring Windows from Python?
Suggestions?
Improvements?

Windows - NagiosPluginsNT - Remote WMI Monitoring

Note to self:

Here is a really good collection of remote WMI monitoring scripts for Windows:

NagiosPluginsNT

It is written in C# and looks nicely done.

December 11, 2008

Big Push 2009 - Free Software Foundation Appeal

About 6 years ago, I spent a good chunk of time as a webmaster for the GNU Project and the Free Software Foundation. Volunteering for the FSF and GNU was one way to express my ideals and somehow give back to the community. This is also what prompts me to release most of my own software under the GPL. The ethics and technical merits of Free Software have appealed to me since I was first introduced to the concept about a decade ago.

Here is the latest newsletter from the FSF regarding their current campaigns. It is a great summary of what is going on in the world of Free Software right now:

The Big Push 2009 -- Free Software Foundation Appeal

"As advocates for free software, we can challenge the status quo and so-called convenience of using the invasive tools of proprietary software companies, because the opportunities for change have never been better."

-Peter Brown
Executive Director, Free Software Foundation

Best Sticker Ever?

best sticker ever?

December 9, 2008

Work Computer Setup

Current Rig - Dec 2008 (AKA "The Matrix"):

... life begins at 3 monitors

Python - Trying Out NetBeans IDE

I normally don't use an IDE for Python development. The majority of my day is spent inside my SciTE editor.

However, Sun recently released NetBeans IDE 6.5 for Python, and I figured I would give it a try. So far I really like it. The Python support works well. It has syntax highlighting, code complete, debugger, integrated console, etc. Overall, it is a comprehensive and nicely designed IDE.

NetBeans is a full featured development environment and supports:
Ajax | C/C++ | Databases | Debugger | Desktop | Editor | Groovy | GUI Builder | Java EE | JavaFX | Java ME | Java SE | JavaScript | Mobile | PHP | Profiler | Python | Refactor | REST | Rich Client Platform | Ruby | SOA | SOAP | UML | Web | WSDL | XML

The IDE is written in Java so I thought it would be dog slow, but it's pretty snappy to work with.

I've decided I will try it for a week and then decide whether to use it as my main dev environment.



Anyone else using NetBeans for Python development?

December 3, 2008

Development Setup: Tools I Can't Live Without

At my current gig, these are the essential tools I use for almost all development:

IDE/Editors:

  • SciTE
  • Visual Studio
  • Eclipse

Language Compilers/Interpreters:

  • Python
  • C#
  • Java
  • Perl

Utilities/Etc:

  • Firefox
  • FileZilla
  • DropBox
  • Cygwin
  • WinMerge

December 2, 2008

"Wolf Fence" Debugging

I heard this term for the first time today, and realized it is a habit I usually follow rather than firing up a debugger and stepping through my code.

Defined by Dennis Lee Bieber here:
http://www.velocityreviews.com/forums/t356093-how-to-debug-python-code.html

"There's one wolf in Alaska, how do you find it? First build a fence down the middle of the state, wait for the wolf to howl, determine which side of the fence it is on. Repeat process on that side only, until you get to the point where you can see the wolf). In other words; put in a few "print" statements until you find the statement that is failing (then maybe work backwords from the "tracks" to find out where the wolf/bug comes from)."

The original term comes from "The Wolf Fence algorithm for debugging" (Edward J. Gauss, ACM - 1982)