Execute External Programs, the Python Ways.

Running external programs are very essential in most programming languages, especially the scripting e.g. BASH, python. For exampe, the well-known grid job scheduler CONDOR can be configured to submit the jobs (which are made of a series of commands) to the scheduler which dispatches the job.

Launching external programs can be classfied by two modes: Synchronous and Asynchronous. The synchronous mode invokes the external commands and waits for the return. The asynchronous mode, on the other hand, returns immediately and continue in the main thread.

In Python, there are many ways to execute external programs. The easiest one is to import the os package. It provides the popen(), system(), startfile() methods.

import os
print os.popen("echo Hello, World!").read()

The os.popen() will treat the output (stdout, stderr) as file object, so you can capture the output of the external programs. It is one of the synchronous methods.

The os.system() is also synchronous, which is fairly easy to use, and it returns the exit-status.

import os
print os.system('notepad.exe')

By acting like double-click in the file explorer, you can use os.startfile() to launch external program that is associated with this file. This is an asynchronous method.

import os
os.startfile('test.txt')

It will throw out an exception if file is not found.

WindowsError: [Error 2] The system cannot find the file specified:

If you install the win32api package (which is not shipped by the python installation by default), you can use the following asynchronous method.

import win32api
try:
    win32api.WinExec('notepad.exe')
except:
    pass

Of course, this is only available on windows platforms. The subprocess package provides a syncrhonous and an asynchronous methods: namely, call and Popen. Both methods take the first parameter as a list. For example,

import subprocess
subprocess.call(['notepad.exe', 'abc.txt'])
subprocess.Popen(['notepad.exe'])
# thread continues ...
p.terminate()

You can use wait() to synchronous the processes. 

import subprocess
p = subprocess.Popen('ls', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
    print line
retval = p.wait()

Another asynchronous method would be to use asynproc package. The example is given below.

import os
from asynproc import Process
myProc = Process("notepad.exe")
while 1:
    # check to see if process has ended
    poll = myProc.wait(os.WNOHANG)
    if poll != None:
        break
    # print any new output
    out = myProc.read()
    if out != "":
        print out

The following is a popular script taken from stackoverflow which shows the usage of subprocess package.

from subprocess import Popen, PIPE
import time

running_procs = [
    Popen(['/usr/bin/my_cmd', '-i %s' % path], stdout=PIPE, stderr=PIPE)
    for path in '/tmp/file0 /tmp/file1 /tmp/file2'.split()]

while running_procs:
    for proc in running_procs:
        retcode = proc.poll()
        if retcode is not None: # Process finished.
            running_procs.remove(proc)
            break
    else: # No process is done, wait a bit and check again.
        time.sleep(.1)
        continue

    # Here, `proc` has finished with return code `retcode`
    if retcode != 0:
        """Error handling."""
    handle_results(proc.stdout)

–EOF (The Ultimate Computing & Technology Blog) —