Blog Archives
Simple script for sftp automation on Windows
The main goal for this script was to automate file transfer from Linux box to Windows Server via sftp. Files saved on Linux server are first transfered to Windows server and after that immediately deleted (purged) from the Linux server.
The script will run once per hour and must satisfy some restrictions that I put in place:
- it must run on any Windows Server 2003 with minimal dependency on third party tools
- it should not depend on WSH or Powershell – a plain old DOS script is a preferred solution in this case
- I was tempted to write script with python but that would cause the script to end under my “support jurisdiction” and I don’t want that to happen ;(. It must be as simple as possible, so that others will not have an excuse not to take over the script for further enhancement – you know who you’re! ;-)
- file transfer must be encrypted using ssh protocol
- no dependency on commercial tools
At first, I considered using Windows port of rsync, cwRsync, but decided not to, mainly because of the cygwin1.dll conflict with existing CopSSH installation and possible with other open source tools that rely on cygwin.
My final decision was to write simple DOS script that’ll depend only on OpenSSH (ssh, sftp). For now, I didn’t put any error notification ability in the script. I’ll probably make a few changes for production version of the script (at least logging, error notification and public key authentication) but for now it is what it is – a draft version .
echo off
:: ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: AlesK
:: Simple script for transfering/purging files with SFTP
:: ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::
:: Global variables
::
set sshuser=oracle
set sshhost=192.168.1.10
set remotedir=/home/oracle/work
set batchcopy=batchcopy.sftp
set batchdelete=batchdelete.sftp
::
::
:: Generate two batch files, one to copy files (get...) from remote host
:: and second one to delete files on remote host (rm...), that'll be
:: used for sftp with option -b <batchfile>
::
::
:: Initialize batch files that are needed by sftp
::
echo # Generated by test.cmd > %Batchcopy%
echo # Generated by test.cmd > %Batchdelete%
::
::
:: After initializing batch files we generate get and rm statements
:: with simple FOR statement.
::
for /F %%i IN ('ssh %sshuser%@%sshhost% 'ls -1 %remotedir%'') DO (
echo get %remotedir%/%%i >> %batchcopy%
echo rm %remotedir%/%%i >> %batchdelete%
)
::
:: So far so good. While writing and testing the script I didn't use public key authentication,
:: instead I chose to enter password each time I run the script.
::
:: At workplace I have an older version of OpenSSH (3.7.p1) that worked flawlesly as such:
::
:: cmd> sftp -b batchcopy.sftp sshuser@sshhost ...and sftp asked me for a password
::
:: At home I'm using newer version of OpenSSH (4.7p1) and the same line of code failed with the error
::
:: Permission denied (publickey,password)
::
:: After some RTFM and Googling (what else?:), I "discovered" that this is a perfectly
:: normal behavior, since sftp is not reading from stdin at all (obviously this was not
:: the case in OpenSSH 3.7).
:: The workaround that I found on the Net is not intuitive at all; with option -o we turn batchmode
:: off immediately before we call batch file with -b. Hmm...what a logic!
::
::
:: Get files with sftp in batch mode (for testing purposes only I prefer password authentication,
:: in production it's better to use public key authentication)
sftp -o "batchmode no" -b %batchcopy% %sshuser%@%sshhost%
::
:: check the errorlevel, if 0 then sftp completed successfully and we can delete files on remote host
if %errorlevel% EQU 0 goto delete
goto end
::
:delete
sftp -o "batchmode no" -b %batchdelete% %sshuser%@%sshhost%
goto end
::
::
:end
echo on
SETSTAMP.BAT – a generic script for “time stamping” file names
From time to time I’m asked how to add timestamp to the file name(s) with nothing more than a standard commands available on every Windows 2000/2003/XP box command line.
You’ll find tons of different recipes on the net, my favorite approach is to use generic script that we can call in our batch files. I attached one such generic batch script to this thread that I’m frequently using in my batch files. You’ll have to customize the script for your local environment, I live in a country with date format DD.MM.YYYY and HH:MI:SS and this is the expected format by the script (FOR statement).
Example of using generic script SETSTAMP.BAT in batch files: :: How to add time stamp to the export log file name :: export-scott.bat :: by calling setstamp.bat we'll set the following variables: :: %DDMMYY%, %DDMMYY-HHMMSS%, %DDMMYY-HHMMSSFF%. call setstamp :: let's timestamp our export dmp/log files... exp scott/tiger@oracle file=scott-%ddmmyy%.dmp log=scott-%ddmmyy%.log :: or use more specific version with time part as well.... :: another call to setstamp will refresh the time call setstamp exp scott/tiger@oracle file=scott-%ddmmyy-hhmmss%.dmp log=scott-%ddmmyy-hhmmss% :: let's display the time the script ended... :: another call to setstamp followed by echo... call setstamp echo %ddmmyy-hhmmss%
SETSTAMP.BAT source code is here:
@echo off
:: Script: setstamp.bat
:: Author: Alesk
:: Generic script for composing timestamp string that you can use to form log file names.
:: Requirement: Windows 2000/XP/2003. Windows NT4 doesn't support date/time system variables,
:: so you'll need to customize the script with date & time commands to make this script work on NT4.
:: Also, you'll need to customize script for your local date format (check with: echo %date% and echo %time%).
:: This script anticipate date format as: Wekday DD.MM.YYYY and for time HH:MM:SS,FF where FF is 1/100 of the second.
::
::
:: Example (test.bat):
:: call setstamp
:: echo Hello!! > mylog%ddmmyy%.log
:: ...
:: call setstamp
:: echo This is second log ... > my2log%ddmmyy-hhmmss%.log
:: Let's go...
:: Get and parse date (in Windows 2000 and later you can use system variables %date% and %time% instead of
:: commands date and time)
:: Customize delims parameter in FOR statement if your date format is delimited in a different way.
FOR /f "tokens=1-4 delims=. " %%i in ("%date%") do (
set dayofweek=%%i
set day=%%j
set month=%%k
set year=%%l)
:: Get and parse system time
:: Customize delims parameter in FOR statement if your time format is delimited in a different way.
FOR /f "tokens=1-4 delims=:," %%i in ("%time%") do (
set hour=%%i
set minute=%%j
set second=%%k
set fraction=%%l)
:: From this point onward you are free to format system variables with basic elements such as day, month, hour...
:: I prefer abbreviation that I can easily remember, here are my three favorite variables:
set ddmmyy=%day%%month%%year%
set ddmmyy-hhmmss=%day%%month%%year%-%hour%%minute%%second%
set ddmmyy-hhmmss-ff=%day%%month%%year%-%hour%%minute%%second%-%fraction%
:: Test section
:: echo %ddmmyy%
:: echo %ddmmyy-hhmmss%
:: echo %ddmmyy-hhmmss-ff%
:: END
Count dates with AWK
Someone on the public forum asked this question:
“Say I want to find out how many dates are there in 1000 lines of characters.Also date format is given (…in my case date format is of ‘DD-MON-YYYY’).”
When I read the question I thought, what a nice exercise for regex with awk. Here is part of the answer I gave:
#######################
# Countdates.awk script
#######################
BEGIN {i=0}
{
$0 = toupper($0)
i += gsub(/(0[1-9]|[1-2][0-9]|3[01])-(JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC)-((19[0-9][0-9])|(20[0-9][0-9]))/,"")
}
END {print "Number of dates (DD-MON-YYYY) in the file: ", i}
#######################
# End Countdates.awk
#######################
Sample file with three lines (dates.txt):
xxxxx01-Jan-200002-Feb-2003 xxxx13-APR-1990xxxxx xx32-OCT-2007---xxx00-dEC-1900 xxx 1-MAY-2003xxxxx xxx2004-AUG-07 xxxxx-08-Aug-2007- 12-Jun-1890
Now, if you run the script:
G:\>gawk -f countdates.awk dates.txt Number of dates (DD-MON-YYYY) in the file: 4</code>
It’ll catch three dates in the first line (01-Jan-2000, 02-Feb-2003,13-APR-1990), none in second line
and one in the third line (08-Aug-2007).
Short explanation of the script:
$0 = toupper($0) ... it's needed to catch Apr, APR, aPR
gsub() functions returns number of substitutions which is added to variable i
/ ...search pattern ... /
(0[1-9]|[1-2][0-9]|3[01]) ... trying to catch only valid days 01-31 (assuming leading zero)
(JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC) .... month names abbreviation in uppercase to compare against input line ($0) which we uppercased at he beggining
((19[0-9][0-9])|(20[0-9][0-9])) .... assuming only years that starts with 19xx and 20xx are valid (that's why we didn't count 12-Jun-1890)
You can take this example and fine “tune” it for the data you actually have. This script lacks true date validation (e.g. it’ll count 29-FEB-2007 as a date despite the fact that it’s really wrong), for this, perl/python etc. are better suited (first catching the date candidates with the power of regex pattern search, then validating those strings as a true dates with some date/time module.).
Python 3 released
Production version of Python 3.0 was released yesterday. It’s a major release that is not compatible with 2.x code. I think it’s a good news for the prospect of the language — it’s better to remove (and replace with something better) some annoyances in the language once for ever. It also means we’ll have to unlearn some things as well as learn some new stuff.
For the start I read Guido van Rossum article about changes: What’s New In Python 3.0.
Other resources I will have to check out while migrating my python scripts from 2.x to 3:
- PEP 3108 – Standard Library Reorganization
- 2to3 – Automated Python 2 to 3 code translation
- Python 3 Documentation
I’ll be watching for a release of third party modules for Python 3, such as cx_Oracle, win32all and Qt and only then I’ll start porting python scripts to Python 3. I guess it’ll take a year, perhaps even more for all third party modules to catch up with Python 3 release.