Category Archives: Scripting
Scripting in general.
VMWare Infrastructure Toolkit for Windows
It was only a question of time when it will be released – a VMWare specific toolkit for Windows PowerShell. I hope I’ll have a time to test this toolkit soon with VMWare Server 2.0 (after it’ll be released); nevertheless, even on paper it sounds promising with 120+ powershell cmdlets centered around managing and automation of VM.
Note: As you might know I was not a particular fan of PowerShell, but I changed my mind :-). I was skeptical about PowerShell when Microsoft announced new scripting environment (what’s wrong with WSH that I’m already familiar with?), but now I understand the importance and potential of this product, it really deserves power in it’s name. If you’re Windows system administrator you’re better off to start learning Powershell right now (if you’re already not) – it’ll make your life much easier and your employment prospect safer;-).
sed one-liners
SED and AWK are my favorites when it comes to (pre)processing data (…before I go and write python script). While my primary source of reference is book titled Sed & Awk, 2nd ed. (O’Reilly), I also have some prints of various sed one-liners hanging on the wall in my office. I never bothered to write down url on which we can find many “popular” sed one-liners, so here it is:
http://sed.sourceforge.net/sed1line.txt
Just an example of how handy this list of one-liners can be. It helped me the other day when someone on the dba-village forum asked about how to load data from comma separated file. There are five fields in the file, but the problem is that the third field may contain any number of commas. (This is a common type of a question that I receive from time to time at my workplace as well — and I almost always resolve for a solution to the sed.)
For example, let’s say that we have file (test.txt) with comma separated data such as this:
COL1,COL2,COL3-with, some, data, with, commas,COL4,COL5
We should enclose field 3 within quotes and then use OPTIONALLY ECNLOSED BY clause in SQL*Loader control file to load the data.
Basically we need to replace second comma with ,’ and next to last comma with ‘, .
The first part is easy:
cmd> cat test.txt | sed "s/,/,'/2" ....will replace second comma with ,'
The second part, replacing next to the last comma with ‘, is more complex and was finally derived from an example written in the one-liner list mentioned above:
cmd> cat test.txt | sed "s/\(.*\)\,\(.*\,\)/\1',\2/"
We're using grouping ( ) and then addressing each of the two groups with 1 and 2. Backslashes are there to escape characters that are part of the syntax, but admittedly making command line string harder for humans to parse. But consider this, lighter example where we want to split days with two consecutive semicolons) :
cmd> echo MondayTuesday | sed "s/\(Monday\)\(Tuesday\)/\1::\2/"
Monday::Tuesday
So that in the end we can get desired result:
cmd> cat test.txt | sed "s/,/,'/2" | sed "s/\(.*\)\,\(.*\,\)/\1',\2/" > test_out.txt
cmd> cat test_out.txt
COL1,COL2,'COL3-with, some, data, with, commas',COL4,COL5
Free PowerShell Commands for Active Directory
Only recently I stumble over Quest “ActiveRoles Management Shell for Active Directory”, a free cmdlets for scripting with AD as a source (and destination). For example, all you need to spool users with their descriptions from AD into csv file is this one liner:
[PS] E:\>Get-QADUser -sizeLimit 0 | select name, Description | Export-Csv ADusers.csv
In addition to “ActiveRoles Management Shell for Active Directory”, you’ll need to install PowerShell itself and Microsoft Core XML Services – if they’re not already installed on your machine (Quest installer will prompt you if that’s is the case).
In “Quest Software” program directory you’ll find link named “ActiveRoles Management Shell for Active Directory” which starts PowerShell.
One problem that I could not find a solution for, is related to spooled file encoding which will loose all non-ASCII characters in my environment (CP1250 for GUI, CP852 for command prompt), even after specifying UTF8 encoding like this:
[PS] E:\>Get-QADUser -sizeLimit 0 | select name, Description | Export-Csv ADusers.csv [PS] E:\>$OutputEncoding BodyName : utf-8 EncodingName : Unicode (UTF-8) HeaderName : utf-8 WebName : utf-8 WindowsCodePage : 1200 IsBrowserDisplay : True IsBrowserSave : True IsMailNewsDisplay : True IsMailNewsSave : True IsSingleByte : False EncoderFallback : System.Text.EncoderReplacementFallback DecoderFallback : System.Text.DecoderReplacementFallback IsReadOnly : True CodePage : 65001
I also tried with console encoding:
[PS] E:\>$OutputEncoding = [console]::OutputEncoding [PS] E:\>$OutputEncoding IsSingleByte : True BodyName : ibm852 EncodingName : Central European (DOS) HeaderName : ibm852 WebName : ibm852 WindowsCodePage : 1250 IsBrowserDisplay : True IsBrowserSave : True IsMailNewsDisplay : False IsMailNewsSave : False EncoderFallback : System.Text.InternalEncoderBestFitFallback DecoderFallback : System.Text.InternalDecoderBestFitFallback IsReadOnly : True CodePage : 852
No luck – in .csv file all “special” characters are lost.
PowerShell columns by Jeffery Hicks
As a long time subscriber on MCP Magazine I also get regular weekly newsletter from them. What I really like to read lately are columns by Jeffery Hicks, introducing the real power of PowerShell with short and simple examples, hence serving me well to learn some basics.
Some direct links to columns by Jeffery Hicks about PowerShell:
- Tee Off
- On File
- PowerShell AKA
- Smooth Operator
- The Two Faces of PowerShell
- PowerShell in Lockdown Mode
- The Big Kahuna
- Member(ship) Has Its Privileges
- Help Is On The Way
- Automated Power for IT
- “Clean” and Mean PowerShell Sessions
- How To Make Windows Spill the Beans
- Jog Your Memory with PowerShell
- Profile of a PowerSheller
- PowerShell Title Prompts
- Look Mom, No Scripting!
- Switching Days in PowerShell
- Color My World
- Put Some Power in Your Event Logs
- PowerShell Pipe Dream
Simple script for sftp automation on Windows
The main goal for this script was to automate file transfer from Linux box to Windows Server via sftp. Files saved on Linux server are first transfered to Windows server and after that immediately deleted (purged) from the Linux server.
The script will run once per hour and must satisfy some restrictions that I put in place:
- it must run on any Windows Server 2003 with minimal dependency on third party tools
- it should not depend on WSH or Powershell – a plain old DOS script is a preferred solution in this case
- I was tempted to write script with python but that would cause the script to end under my “support jurisdiction” and I don’t want that to happen ;(. It must be as simple as possible, so that others will not have an excuse not to take over the script for further enhancement – you know who you’re! ;-)
- file transfer must be encrypted using ssh protocol
- no dependency on commercial tools
At first, I considered using Windows port of rsync, cwRsync, but decided not to, mainly because of the cygwin1.dll conflict with existing CopSSH installation and possible with other open source tools that rely on cygwin.
My final decision was to write simple DOS script that’ll depend only on OpenSSH (ssh, sftp). For now, I didn’t put any error notification ability in the script. I’ll probably make a few changes for production version of the script (at least logging, error notification and public key authentication) but for now it is what it is – a draft version .
echo off
:: ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: AlesK
:: Simple script for transfering/purging files with SFTP
:: ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::
:: Global variables
::
set sshuser=oracle
set sshhost=192.168.1.10
set remotedir=/home/oracle/work
set batchcopy=batchcopy.sftp
set batchdelete=batchdelete.sftp
::
::
:: Generate two batch files, one to copy files (get...) from remote host
:: and second one to delete files on remote host (rm...), that'll be
:: used for sftp with option -b <batchfile>
::
::
:: Initialize batch files that are needed by sftp
::
echo # Generated by test.cmd > %Batchcopy%
echo # Generated by test.cmd > %Batchdelete%
::
::
:: After initializing batch files we generate get and rm statements
:: with simple FOR statement.
::
for /F %%i IN ('ssh %sshuser%@%sshhost% 'ls -1 %remotedir%'') DO (
echo get %remotedir%/%%i >> %batchcopy%
echo rm %remotedir%/%%i >> %batchdelete%
)
::
:: So far so good. While writing and testing the script I didn't use public key authentication,
:: instead I chose to enter password each time I run the script.
::
:: At workplace I have an older version of OpenSSH (3.7.p1) that worked flawlesly as such:
::
:: cmd> sftp -b batchcopy.sftp sshuser@sshhost ...and sftp asked me for a password
::
:: At home I'm using newer version of OpenSSH (4.7p1) and the same line of code failed with the error
::
:: Permission denied (publickey,password)
::
:: After some RTFM and Googling (what else?:), I "discovered" that this is a perfectly
:: normal behavior, since sftp is not reading from stdin at all (obviously this was not
:: the case in OpenSSH 3.7).
:: The workaround that I found on the Net is not intuitive at all; with option -o we turn batchmode
:: off immediately before we call batch file with -b. Hmm...what a logic!
::
::
:: Get files with sftp in batch mode (for testing purposes only I prefer password authentication,
:: in production it's better to use public key authentication)
sftp -o "batchmode no" -b %batchcopy% %sshuser%@%sshhost%
::
:: check the errorlevel, if 0 then sftp completed successfully and we can delete files on remote host
if %errorlevel% EQU 0 goto delete
goto end
::
:delete
sftp -o "batchmode no" -b %batchdelete% %sshuser%@%sshhost%
goto end
::
::
:end
echo on
dbaportal.eu 