Batch scripts...

While working on a project recently I needed to find a solution to transfer all files from two separate newswire provider FTP sites into a customer's news system on an automated basis. Unfortunately, the Windows-based server designed to ingest the news feeds (which were xml files) couldn't have an FTP site set as its watch folder, which was a limitation of the software, not the OS. 

I decided to create a batch file for synchronising the files from each FTP site into local folders that the news ingest software was watching. The news ingest software performs its own housekeeping by deleting the files from the watch folders once they have been ingested into the newsroom database, preventing the volume from filling up.

To make things even more interesting, the customer's login to one of the remote FTP sites had read-only permissions!

This mattered greatly because files could easily end up being ingested into the news database multiple times without the ability to delete files from the remote FTP site once they had been ingested into the news database.

To complicate matters further, a second newsroom system also needed to ingest the same stories as the one I was working on. This wasn't a problem with regards to the read-only FTP site but for the system that we could delete from, in order to make sure we didn't ingest files multiple times, it was a problem that had to be taken into a consideration because it meant that we had to ensure that files made it to both newsroom systems before they were deleted from the source FTP folder.

The final issue was that the second newsroom system could only watch FTP folders, not network shares!

I used a combination of WinSCP, Windows Task Scheduler and FileZilla server as the solution I put in place.

For the FTP folder that we had permission to delete from, I used the following script:

WinSCP.exe /command "option batch abort" "open ftp://user:password@remote.ftpserver/" "get *.xml c:\watchfolder1\" "get -delete *.xml c:\second-news-system\" "exit"

This script downloaded the files from the remote FTP site to two folders. Then once the second copy had completed, the source files are deleted, preventing duplicated downloads when the script is re-run. The folder for the second news system was shared over FTP using Filezilla Server.

For the FTP folder that we had read-only permissions on, I had to use two scripts. One to download all files on the remote FTP site to a local "mirror" folder, and another to subsequently copy the files into the local watch folder.

We couldn't copy directly into the watch folder because all WinSCP can do is compare files in the source and destination folders based on size and modification timestamp. Without the ability to delete from the source FTP folder, once the news ingest software had purged files from the watch folder WinSCP would simply download the same files again, resulting in duplicate ingests into the news system.

One idea was to set WinSCP to only download files less than a minute old and run the script every minute. The problem with that was that the remote FTP site was updated with files that had modification timestamps that were sometimes 10 minutes old. WinSCP is able to update timestamps though so I took advantage of that feature.

I finally used this script to initially mirror the remote read-only FTP folder to a local folder that could then b:

WinSCP.exe /command "option batch abort" "open ftp://user:password@remote.ftpserver/" "synchronize local -criteria=size -nopreservetime -filemask=*.xml C:\FTP-Mirror\ /" "exit"

Notice that by using the -nopreservetime option, the modification timestamps are updated.

Due to WinSCP not being able to move files between local folders, I then used FileZilla Server to enable me to connect to it via FTP and move files from it to the relevant news system watch folder.

WinSCP.exe /command "option batch abort" "open ftp://user:password@localhost/" "synchronize local -filemask=*.xml>=1N C:\watchfolder2\ /" "exit"

This script copied files that were newer than 1 minute and I set it to run every minute.

The final thing to do was to put a script in place to delete old files from the FTP-Mirror folder. I wanted to run a script once a day at midnight and delete files that were older than 2 days so I used the following script:

forfiles /p "C:\ftp-mirror" /s /m *.* /c "cmd /c Del @path" /d -2

That's it. The 4 batch files are all automated by Windows Task Scheduler and have been working away happily for a while.

Ping a range of IP addresses from a PC without installing anything

While measuring round-trip times between sites separated by virtual ethernet circuit the other day, I wanted to ping an entire 24-bit IP subnet. 

"Easy" I hear you cry! Just run AngryIPScanner or similar and off you go!

Unfortunately installing or running third-party software on this particular server wasn't permitted so faced with the prospect of running the ping command endlessly (well, over 250 times) I decided to see if there was another way of achieving my end goal that only involved running one command. 

It turns out there's a way to do it!

It's possible to run a for loop from the Windows command line using the FOR command.

Better still, you can use the for command to loop through a range of numbers using FOR /L followed by the relevant parameters.

The syntax is "FOR /L %%parameter IN (start,step,end) DO command" and so you just need to substitute the values for whatever you need.

My start, step and end parameters were 1, 1, and 254 since I wanted to ping all IP addresses from 1 to 254 on the subnet.

The command I wanted to run was ping and I only wanted to ping each IP address once so I used the -n switch. I also specified a timeout value of 300ms using -w. If all the hosts on your subnet are located in the same facility you could reduce that timeout value down to something like 50ms.

Finally I output the results to a text file called ping_subnet.txt to make reading over them easier.

The final command looked like this:

FOR /L %G in (1,1,254) DO ping 192.168.10.%G -n 1 -w 300 >> ping_subnet.txt

Remotely enable VNC access to a Mac

If you are trying to connect to the desktop of a Mac from a Windows machine using a VNC client it should all work happily. 

However, if the Mac's admin user has disabled VNC access in the screensharing settings or if the Remote Desktop service has stopped on the Mac, here's what you can do to re-enable it:

1. Connect to the Mac using Putty and SSH. Be sure to login as a user with admin rights.

2. Run the following command, substituting the MyVNCPassword for your own password:

sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -activate -configure -access -on -clientopts -setvnclegacy -vnclegacy yes -clientopts -setvncpw -vncpw MyVNCPassword -restart -agent -privs -all

3.  Try to connect to the Mac from your VNC client. You will be prompted for the password you declared in the command. 

4. You should now have a VNC remote desktop session open to the Mac from the PC!

The moral of this story is that if you need to manage a network of Macs from a PC on the same network, make sure you enable SSH remote access on them!

OnyX Utility for Mac

OnyX is a great tool for maintenance and tweaking of the Mac OS. Here are 5 things that OnyX can help with:

1. Checking the S.M.A.R.T. status of your system drive. The purpose of S.M.A.R.T. (self-monitoring, analysis and reporting technology) is to attempt to anticipate hard drive failures. Regularly checking the status can save you a whole lot of heartache by alerting you before your system drive fails. OnyX allows you to quickly check the SMART status to give you some peace of mind.

2. When you customise the OS or an individual app, these settings are saved in preference files. Trashing preference files has been a standard procedure when troubleshooting apps on the Mac OS for a long time. OnyX will read the syntax of all the preference files on your system and tell you if any need to be trashed.

3. The associations between documents and applications are stored within the LaunchServices database. As you install and uninstall apps, this database can become out of date or contain duplicate entries. Fear not, OnyX allows you to rebuild the LaunchServices database.

4. If you're setting up a limited user account you can use OnyX to turn off many of the menu options that are usually available within Finder such as 'Preferences' or 'Empty Trash'.

5. You can set a welcome message or display an acceptable usage policy at the login window.

If you want to run a number of maintenance tasks in one go then you can even automate multiple tasks to run sequentially!

OnyX has been freely available since 2005 for OS 10.2 and above. You can download an appropriate version from Titanium Software http://www.titanium.free.fr/pgs/english/apps.html

Is it a Nehalem or a Harpertown?

A quick way to tell MacPro models apart is to look at the Model Identifier. To view this,

1. Click on 'About this Mac' from the Apple menu
2. Click 'More Info' to open the System Profiler
3. The Model Identifier is located in the Hardware Overview page and should look something like this:

What do these identifiers mean? Here's a list for the MacPro:

• MacPro 1,1 and Mac Pro 2,1 = original models
• MacPro 3,1 = Intel Harpertown, Early 2008 model
• MacPro 4,1 = Intel Xeon Nehalem model
  

Can you install a USB webcam?

"Have you tried plugging it in to see what happens?" was my response. Apparently the person with the problem had tried this so I had a go.

After a google search I found the solution to the issue in a blog entry from a guy who works for Logitech support.

INF files are text files that contain setup information that tells Windows where to copy files and make registry changes during the installation of device driver or other software. On Windows XP, they are found in %systemroot%\inf. On this particular computer - and 19 others that had been loaded from the same image - the usb.inf, file was missing.

Luckily, the i386 folder was on the system drive with a compressed version of the file located within. Following the instructions from the page linked above worked a treat.

Assuming your i386 folder is at the root of dive C:, type the following at the command prompt to expand the file from the i386 folder:

cd C:\i386
expand usb.in_ c:\windows\inf\usb.inf
expand input.in_ c:\windows\inf\input.inf
expand wdma_usb.in_ c:\windows\inf\wdma_usb.inf

After a restart, the USB device will install as expected.

This was a strange fault and I still have no idea why the file was missing in the first place!

Controlling a Mac using VNC on a PC

This should be straightforward now that OS X has VNC built-in (screen sharing) but there are a couple of things you need to remember when attempting to get this working.

First of all, you need to ensure that screen sharing is enabled. Go to System Preferences > Sharing and enable Screen Sharing. You also need to set a password so open the computer settings and set something sensible.

The next thing you need to remember is that OS X doesn't accept VNC connections that request 8-bit colour. If you attempt to connect to a Mac using Real VNC viewer running on a PC, configure the Colour level options beforehand so that you're using 'Full' or 'Medium' and you should be fine.