Category Archives: Scripting

Notes related to scripting/programming techniques and code snippets (python, awk, sed, go, powershell …) will be published here.

IronPython – how to compile exe

IronPython 2.6 ships with pyc.py (look in %IRONPYTONINSTALLDIR%\Tools\Scripts\pyc.py), script that can be used to compile python script(s) into binary (exe). For the purpose of this mini how-to I created a directory D:\IronTestCompile with the following contents:

D:\IronTestCompile>dir /b /oe
Microsoft.Scripting.ExtensionAttribute.dll
Microsoft.Scripting.dll
Microsoft.Scripting.Core.dll
IronPython.dll
IronPython.Modules.dll
Microsoft.Dynamic.dll
hello2.py
hello1.py
pyc.py
HelloWorld.py

From IronPython directory I copied all the necessary DLL’s (Microsoft*.dll, IronPython*.dll) that are needed for compilation and as run-time prerequisite to run compiled HelloWorld.exe. HelloWorld.py is my main script that imports two additional modules, hello1.py and hello2.py and pyc.py is compilation script shipped with IronPython 2.6.

Here is the contents of HelloWorld.py, hello1.py and hello2.py:

#
# HelloWorld.py
# IronPython - Compile test
#
import hello1
import hello2

print "HelloWorld from HelloWorld.py"
hello1.Hello()
hello2.Hello()
# End

# hello1.py
def Hello():
    print "Hello World from Hello1.py"

# hello2.py
def Hello():
    print "Hello World from Hello2.py"

Now let’s see what we can do with pyc.py script:

D:\IronTestCompile>ipy pyc.py

pyc: The Command-Line Python Compiler

Usage: ipy.exe pyc.py [options] file [file ...]

Options:
    /out:output_file                          Output file name (default is main_file.<extenstion>)
    /target:dll                               Compile only into dll.  Default
    /target:exe                               Generate console executable stub for startup in addition to dll.
    /target:winexe                            Generate windows executable stub for startup in addition to dll.
    /? /h                                     This message

EXE/WinEXE specific options:
    /main:main_file.py                        Main file of the project (module to be executed first)
    /platform:x86                             Compile for x86 only
    /platform:x64                             Compile for x64 only


Example:
    ipy.exe pyc.py /main:Program.py Form.py /target:winexe

And here is my command line that I used to compile console application HelloWorld.exe:

D:\IronTestCompile>ipy pyc.py hello1.py hello2.py /out:HelloWorld /main:HelloWorld.py /target:exe
Input Files:
        hello1.py
        hello2.py
Output:
        HelloWorld
Target:
        ConsoleApplication
Platform:
        ILOnly
Machine:
        I386
Compiling...
Saved to HelloWorld

End result of compilation are two files, HelloWorld.exe and HelloWorld.dll. Both files (plus above listes DLL’s) are needed for application to run. And here is sample output:

D:\IronTestCompile>HelloWorld
HelloWorld from HelloWorld.py
Hello World from Hello1.py
Hello World from Hello2.py

How to compile cx_Oracle module on Windows x86

Usually I would not touch with a stick open source package that need to be compiled on Windows. cx_Oracle is different. For me it’s a mandatory python extension module that just need to be there when I need to access Oracle database from python script. Until recently, I was happy with more or less promptly released binaries. They always installed and worked flawlessly. The problem is that at the time of this writing I’m starting to evaluate python 3.1.1 for which cx_Oracle is not yet available. It’s a trivial task to compile cx_Oracle on Linux, but I need Windows version as well. So, for the first time I tried to compile cx_Oracle on Windows by myself.

According to author build instructions, he recommends MinGW compiler for builds on Windows. To make my story short, I downloaded MinGW and to the best of my knowledge tried to compile cx_Oracle module. I miserably failed, each time with an error stating that vcvarsall.bat could not be found. Obviously setup.py script or perhaps python distutils module itself simply expects Microsoft compiler, Visual C++ to be present. After spending a couple of hours searching the web for hints I decided to try another route, with Microsoft Visual Studio Express 2008. Surprisingly cx_Oracle 5.0.2 compiled easily.

Here are the steps that I followed:

  • downloaded cx_Oracle source code (see link “Source code only”) and unpacked the tar in temporary directory
  • downloaded and installed Microsoft Visual C++ 2008 Express edition.
  • downloaded and installed Python 3.1.1
  • downloaded and installed Oracle10g client (10.2.0.4)
  • compiled and installed cx_Oracle:
  • 1) Open Visual Sudio Command Promt, Start -> Run:

    %comspec% /k ""C:\Program Files\Microsoft Visual Studio 9.0\VC\vcvarsall.bat"" x86

    2) within VS 2008 command prompt cd to directory where you untared cx_Oracle

    C:\Program Files\Microsoft Visual Studio 9.0\VC> cd c:\Python3\cx_Oracle-5.0.2
    C:\Python3\cx_Oracle-5.0.2>

    3) set environment, for example:

    C:\Python3\cx_Oracle-5.0.2> SET ORACLE_HOME=D:\ORACLE\ORA10
    C:\Python3\cx_Oracle-5.0.2> SET PATH=C:\PYTHON3;%ORACLE_HOME%\BIN;%PATH%

    4) compile and install cx_Oracle

    C:\Python3\cx_Oracle-5.0.2> python setup.py build
    C:\Python3\cx_Oracle-5.0.2> python setup.py install

    5) test cx_Oracle by trying to import module

    C:\Python3>python
    Python 3.1.1 (r311:74483, Aug 17 2009, 17:02:12) [MSC v.1500 32 bit (Intel)] on
    win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import cx_Oracle
    >>>

Visual Studio 2008 Express is free but need to be registered within 30 days, registration is completely up to you, if your only goal was to compile cx_Oracle you don’t need registration.

Twitting with Python

I would like to start this note with a fair disclaimer to all those who might know me and my stand about “social networking” that cropt in recent years:


“The only reason why I finally surrendered to so-called micro-blogging is my new phone. That does not mean that my opinion about “twittering” has changed a bit: I still feel pity for all those dummies out there that “follows” others on twitter. Only Facebook sheeps ranks lower in my book :-). That said, I do recognize that micro-blogging sites with well documented API might be useful for pure technical utilization. That’s why I opened account on Twitter that will serve to me and only me. Instead of sending alerts to my phone via SMS messages I’ll write a simple python script that will post “statuses” to my twitter account via API. And since my HTC Hero comes with configured Peeps, client application for Twitter, I decided to give it a try, so I opened “non-public” twitter account where python script will send various non-critical technical alerts. Of course, messages that I’m intending to send to twitter.com will be so obscure that only I and perhaps a couple of my associates will be able to put in the context. Not a chance that any of the messages could jeopardize my client security in any way, even if someone unauthorized could see them.


Enough rumblings, let’s see how easy it is to exploit twitter.com for some productive work.

With some help from the Google I decided to use python-twitter module.

Steps to install python-twitter and a prerequisite, simplejson:

1) download and install appropriate version of setuptools from pipy.python.org.

I downloaded and installed setup tools for python 2.6 win32..

This is a prerequisite for simplejson (setuptools version >= 0.6c7).

2) download and install simplejson

– after extracting simplejson-2.0.9.tar.gz to some temporary directory:

cmd> cd simplejson-2.0.9
cmd> python setup.py install

3) download and install Python Twitter

– after extracting python-twitter-0.6.tar.gz to some temporary directory:

cmd> cd python-twitter-0.6
cmd> python setup.py build
cmd> python setup.py install
cmd> python setup.py test

4) read documentation about twitter module API

5) let’s see python-twitter module at work:

#
# Simple test case to check if python twitter module works.
# 
import twitter

# Twitter login account and password
tusername = 'dbatwitaccount'
tpassword = 'mysecret'

# Initialize authenticated twitter API session
api = twitter.Api(username=tusername,password=tpassword)

def PostTweet(message):
    """
    Function for posting a message. 
    """
    if len(message) < 141:
        status = api.PostUpdate(message)
    else:
        print ('Error: message is longer than 140 characters!')

def PrintUserTweets(tusername, cnt=0):
    """
    Simple function for printing tweets. You can limit the
    number of tweets with the cnt parameter, default is 0
    which means get and print all tweets.
    """
    statuses = api.GetUserTimeline(user=tusername, count=cnt)
    for status in statuses:
        print (status.created_at + ': [' + status.text + ']')

def DeleteAllTweets(tusername):
    """
    Function for deleting all tweets - use with care!
    """
    i = 0
    api.SetCache(None)
    statuses = api.GetUserTimeline(user=tusername, count=0)
    for status in statuses:
        api.DestroyStatus(status.id)
        i = i + 1
    print ("Number of deleted tweets: " + str(i))

# posting a tweet -- in real life scenario DBA should
# be careful to post only obscure enough messages,
# not to reveal such details as SID, service names,
# hostnames, IP's, account names, even errors...
# The example below is only an example, in real-life
# I would not post ORA- message number nor DB name!
PostTweet('ORA-600 @ DB1 / check alert.log /')
               
# print all user tweets
PrintUserTweets(tusername)

# print limited number of tweets (for example last two)
PrintUserTweets(tusername, 2)

# delete all tweets
DeleteAllTweets(tusername)

# End

With above script I only glimpsed over twitter API that is exposed through python twitter module. Refer to documentation for a complete overview.

Finding last Logon time with PowerShell

I needed to find out last AD log-on time for a particular Active Directory user account on all our domain controllers. I already knew that it should not be particular difficult to come up with some PowerShell one-liner.

With some help aside by the Google, I finally came up with the statement that I liked and that is worthy of this note for my future reference:

PS>
PS>
PS> $username='alesk'
PS> Get-QADComputer -ComputerRole DomainController | foreach {(Get-QADUser -Service $_.Name -SamAccountName $username) | select Name, DisplayName, LastLogon, Path} | sort LastLogon
PS>

…and with the output similar to this one:

Name DisplayName LastLogon Path
---- ----------- --------- ----
alesk ales-k 07.08.2009 23:30:00 LDAP://acme-dc2.corp.com...
alesk ales-k 10.08.2009 11:21:57 LDAP://acme-dc1.corp.com...
alesk ales-k 10.08.2009 11:40:01 LDAP://acme-dc3.corp.com...
alesk ales-k 10.08.2009 15:20:25 LDAP://acme-dc4.corp.com...

The logic behind the script is simple:

1) get the list of all the DC’s from ActiveDirectory (Get-QADComputer -ComputerRole DomainController).

2) then query each domain controller (-Service $_Name) for the account passed as variable ($username) and then select the attributes that are needed with ith the select statement (use select * to examine the vast amount of attributes available for querying).

3) finally, I wanted the result to be sorted by LastLogon field.

Why Python?

The other day I was reading an August issue of my favorite magazine, Linux Journal. On page 17 there is a list of the ten most popular articles off all-time on LinuxJournal.com. At the #1 is article written by Eric Raymond, titled Why Python?.

I was pleasantly surprised. It’s precisely this article that ended my ultimate quest for scripting language back in 2004. Frankly, I was sick of Microsoft WSH that I was using back then; besides I needed scripting language that could run equally well on Windows & Linux (support for other major flavors of Unix would be a plus) and with solid support for various text processing/parsing functions, that could allow me to “glue” together some tasks related to ETL (mostly DBA stuff).

I spent considerable amount of the first half of year 2004 evaluating various scripting languages — somehow skipping python at first (can’t remember why I dismissed python so quickly). Since I knew that Perl has strong parsing capabilities (regular expressions etc.), it was this language that I used to “benchmark” all others against.

Perl could probably pass my evaluation if only the written code would not look so wacky (with all due respect to Perl community) – then one day Google returned link to the article written by Eric Raymond, “Why Python?”. It was revealing experience reading this article. I downloaded and installed python immediately after I finished with the article, browsed the manuals and when I saw some code examples, I thought: what a beauty (sorry if it sounds too geeky)! It was love at first sight, I knew I was hooked for life. Since then, I never looked back, python is my favorite scripting language. The only thing I really regret is that I didn’t “find” the python earlier in my career. What a pity!