Multithreaded Programming w/ Delphi 5 on Windows XP

I have successfully solved a very nasty little programming problem which only appears to show up under Windows XP. First, a little backround: At my work, we provide loan calculation libraries to other parties, for use in their systems. One of our potential customers requested that we prove that our 32-bit Windows library is thread safe.

I knew that the library (written in Delphi 5) was currently not thread safe (it wasn't written by me!), but estimated that I could get it up to snuff in about a day or two. Before 'fixing' the library, I set out first of all to write a multithreaded client to prove that it was not thread safe. That way, I could then fix the DLL and then re-run the tests, hopefully proving that what was once broken was now fixed, etc.

Delphi is a great language to program in. It is object oriented, with full support for all the OOP buzzwords like: polymorphism, information hiding, etc. Couple that with a great support library and a lightning quick compiler, and you have a very great combination. So, I quickly whipped up an application that spawns 4 worker threads, each of which compute a different loan 4000 times. Each time through the calculation loop, it checks to see if the results returned from the thread-unsafe library have changed from the correct, base line results computed in a single thread at application initialization.

As expected, the app crashes hard with EInvalidPointer exceptions, etc. It is ugly indeed. So, I then go into the DLL and strategically place some critical sections to protect the vulnerable areas, and then run the multithreaded test again. And it still doesn't work. However, it DOES work on Windows boxes of the flavors 98 and 2000. Very odd.

It is hear that my 40 days in the programming wilderness begins. I write a second, simpler console based multithreaded client in the thoughts that perhaps the vcl thread was creating problems. That didn't do the trick. I simplified the test apps as much as possible, with no luck.

After three days of pounding my head against this programming brick wall, scouring the net via Google, I stumbled upon a reference to a global variable names IsMultiThread, supposedly set up by Delphi in System.pas. This variable is supposed to be automatically set to true whenever you use Delphi's TThread object, and one would guess that it should also be set for DLL projects. I decided to explicitly set this variable to True in the initialization section of the DLL, as well as in the test applications startup.

EUREKA! The problem was solved, and I was able to rest over the weekend. Except that 2 out of 3 kids were sick, but that is an entirely different saga. So, the moral of the story is that when writing a multithreaded library or application in Delphi 5, just go ahead and explicitly set IsMultiThread := true. I hope this post can help someone avoid the same problem again!

Google Maps!

Google Maps has just opened its doors as of today. Currently, it it fairly javascript heavy and only works well in Mozilla browsers (including Firefox) and Internet Exploder. Looks pretty slick, at least as far as my brief test can tell.

Hotplug Support for Everyone in Debian Sarge

Over my vacation, I put together a new system at home consisting of: AMD Sempron 2500+, 512MB ram, 120MB 7200rpm drive, old GeForce2MX graphics card, and a new case w/ USB ports in the front. I decided to set it up in a dual boot configuration: Windows XP for gaming, and Debian Linux for everything else.

Debian's new installer project is paying off with a very slick system. The only thing I didn't realize until after the installation was that the default kernel to install was 2.4. There is a boot option to install a 2.6 kernel, but I missed it. No biggie, a simple "apt-get install kernel-image-2.6-k7" solved that particular problem, and everything seemed to be running smooth as silk. Sound, networking, etc. - everything just worked. Very nice, and an easier install than Windows XP.

However, I did have one stumbling block which I had to overcome. For some reason, I couldn't get the gnome desktop to recognize my USB mass storage device once it was plugged in. Checking the output from dmesg showed that it was being correctly recognized and set up, so it must have been something else. Logging in as root, I descovered that it worked fine. This fact zeroes the problem down to one of permissions. There is a permission that normal users on the system need to have in order to be able to mount the flash drive in question.

After looking at the /etc/group file, I saw a group named 'plugdev', which appeared promising. Using Google (aka the index to the internet), I verified that this was a likely cause to my problems, and added my 'normal' user account to the group. Bang - zoom, the problem was solved!

And so I write this post, hoping that I might be able to help someone else out having the same problem I did, and at the same time recording it down for myself in case of future use!

Free Antivirus Software for the Home?

Many people spend way too much money for antivirus software which some companies turn into bloated, resource draining suites. If you want to try something different, give Avast! Home Edition a look. It is completely free for home, non-commercial use.

Progeny Debian Developer Edition RC1

My Debian SID (still in development) installation on my home PC has been acting up for a while now, but not enough to force me to reinstall and go through the whole process of setting things up just right. A few days ago, I checked out the SUSE Linux 9.2 live CD, and was fairly impressed with it. Unfortunately, Novell/SUSE has not yet placed the full distribution on their ftp servers, so I began looking elsewhere.

The next day, I saw that OSDir.com had a brief article with an extensive screenshot gallery posted showing Progeny's Debian Developer Edition, Release Candidate 1. Since Debian is probably my favorite distribution I've used to date, I figured that I would give it a shot and proceeded to download the two CD images containing the required installation files.

To make a long story shorter, I deleted all files on my existing linux partition except for the /home and /etc directories, and renamed them /home_old and /etc_old. I kept the home directory since it contains all of the user data and files, whereas /etc contains lots of setup information that I thought I might need after the installation.

The 2 CD installation proceeded smoothly, although the actual process of copying the package contents to the system seemed to take a while. This could be due to the CD-R media the iso images were written upon, but I'm not sure. After the installation was completed, I rebooted the system and ... nothing. Crap!

I messed around with the grub boot loader options, and nothing I tried seemed to improve matters. I then moved my hard drive from /dev/hde to /dev/hdc, thinking that perhaps the kernel was having problems accessing ide3. After tweaking grub to try and load the kernel from /dev/hdc and removing the silent boot option, I got a crc error in the initial stages of loading the kernel. So, I figured that I would try to install it one more time.

Roughly 1.5 hours later, the system was installed (again), and the moment of truth arrived as the machine rebooted. This time, the system came up with no detectable problems! Woo-hoo!

My initial thoughts on the system after a paltry hour of use is that the distribution is clean, functional, and nice. I installed the workstation version to get all of the development tools installed along with the normal desktop apps. Gnome is the desktop environment installed, with Evolution as the mail client, Firefox RC1 as the web browser, and OpenOffice 1.2 as the office productivity suite. As a first impression of the distribution, I'd have to give it two thumbs up. I am sure that as I use it more, my opinion will be refined depending upon what problems/gems I encounter.

Exporting Your Evolution Address Book for Gmail

When logging onto my Gmail account today, I noticed a new, highlighted link near the top of the screen titled “New Features!”. After investigating these new features, I discovered that Gmail now allows you to import contact information via an uploaded CSV (comma seperated value) file. Great!

I use Evolution on my Gnome Linux desktop to manage my email and contacts. Looking through the various menus, I realized that there is no way (which I could see) to export my contacts file to a CSV file. Ug! However, since Evolution is Open Source software, that meant that the file format for the address database was not proprietary, and hence that someone else out there had probably already solved this problem. So, doing a Google search for exporting evolution contacts to CSV files, and I stumble upon this page which provides a python and perl script to accomplish the task. I downloaded the scripts and crossed my fingers...

Hmmmm. No joy. It was time to get my hands a bit dirty and mingle with the code, to try and figure out what was going on. The python script worked fine once I made sure that /usr/env/python2 pointed to the latest version of python installed on my system (version 2.3) instead of version 2.0, which it was stuck on. It turns out that Evolution stores the contact information using the Berkely DB (version 7) format. The python script opens up the evolution contact database (addressbook.db), and then outputs the contact information in VCARD format to standard output.

Here is the python script, named exportevolution.py:

#!/usr/bin/python2

import bsddb, os, re

home = os.getenv('HOME')

# This is the only variable you should change.  If you do not know the
# location of your evloution address book, use `locate addressbook.db`
# to find it

dbname = '%s/evolution/local/Contacts/addressbook.db' % home

db = bsddb.hashopen(dbname)

for k in db.keys():
    for line in db[k].split('\n'):
        print line

db.close()

Once I had the python script working, the perl script was next on software TO-DO list. The perl script takes input from standard input, and parses the lines fed to it looking for specific VCARD tags which evolution uses. It then interprets these tags, and output them in a CSV format. Sort of. After quickly acquainting myself with basic Perl, I dove into the script, generalizing it by adding constants to specify the separator for each field, and adding quotes around each field. I cleaned up a few other small problems so that it would work in Perl's strict mode, and tested it out. Success!

Here is the perl script, named vcard2csv.pl:

#!/usr/bin/perl

use strict;
my $begin;
my $end;
my $quo;
my $sep;
my $line;
my $name;
my $tel_work_voice;
my $tel_work_fax;
my $tel_home;
my $tel_pager;
my $tel_cell;
my $tel_voice;
my $email;

# initialize constants
$begin = "BEGIN:VCARD";
$end   = "END:VCARD";
$quo   = '"';
$sep   = ",";

# print out the header for the CSV file
print $quo . "Name" . $quo . $sep;
print $quo . "Work - Voice".$quo . $sep;
print $quo . "Work - Fax".$quo . $sep;
print $quo . "Home".$quo . $sep;
print $quo . "Pager".$quo . $sep;
print $quo . "Cell".$quo . $sep;
print $quo . "Voice".$quo . $sep;
print $quo . "E-mail".$quo;
print "\n";

while ( $line = <STDIN> ) {
    chomp ($line);
    chop ($line) ; # trailing ^M

#BEGIN:VCARD
    if  ( $line eq $begin ) {

        $name = "";
        $tel_work_voice = "";   
        $tel_work_fax = "";
        $tel_home = "";
        $tel_pager = "";
        $tel_cell = "";
        $tel_voice = "";
        $email = "";

    }

#X-EVOLUTION-FILE-AS:
    if ( $line =~ /^X-EVOLUTION-FILE-AS\:.*/ ) {

        $line =~ s/^X-EVOLUTION-FILE-AS://g;       
        $name = $line;
        print $quo . $name . $quo . $sep;

    }

#TEL;WORK;VOICE;
    if ( $line =~ /^TEL;WORK;VOICE:.*/ ) {

        $line =~ s/^TEL;WORK;VOICE://g;
        $tel_work_voice = $line;

    }

#TEL;WORK;FAX;
    if ( $line =~ /^TEL;WORK;FAX:.*/ ) {

        $line =~ s/^TEL;WORK;FAX://g;
        $tel_work_fax = $line;

    }

#TEL;HOME;
    if ( $line =~ /^TEL;HOME:.*/ ) {

        $line =~ s/^TEL;HOME://g;
        $tel_home = $line;

    }

#TEL;PAGER;
    if ( $line =~ /^TEL;PAGER:.*/ ) {

        $line =~ s/^TEL;PAGER://g;
        $tel_pager = $line;

    }

#TEL;CELL;
    if ( $line =~ /^TEL;CELL:.*/ ) {

        $line =~ s/^TEL;CELL://g;
        $tel_cell = $line;

    }

#TEL;VOICE;
    if ( $line =~ /^TEL;VOICE:.*/ ) {

        $line =~ s/^TEL;VOICE://g;
        $tel_voice = $line;

    }

#EMAIL;INTERNET;
    if ( $line =~ /^EMAIL;INTERNET:.*/ ) {

        $line =~ s/^EMAIL;INTERNET://g;
        $email = $email . $line;

    }

#EMAIL;QUOTED-PRINTABLE;INTERNET;
    if ( $line =~ /^EMAIL;QUOTED-PRINTABLE;INTERNET:.*/ ) {

        $line =~ s/^EMAIL;QUOTED-PRINTABLE;INTERNET://g;
        $line =~ s/=0A/ /g; #=0A make it a space
        $email = $email . $line;

    }

#END:VCARD   
    if ( $line eq $end ) {

        if ( $tel_work_voice eq "" )
        { print $sep;}
        else
        { print $quo . $tel_work_voice . $quo . $sep; }
        if ( $tel_work_fax eq "" )
        { print $sep; }
        else
        {print $quo . $tel_work_fax . $quo . $sep; }
        if ( $tel_home eq "" )
        { print $sep; }
        else
        {  print $quo . $tel_home . $quo . $sep; }
        if ( $tel_pager eq "" )
        { print $sep; }
        else {  print $quo . $tel_pager . $quo . $sep; }
        if ( $tel_cell eq "" )
        { print $sep; }
        else { print $quo . $tel_cell . $quo . $sep; }
        if ( $tel_voice eq "" )
        { print $sep; }
        else
        { print $quo . $tel_voice . $quo . $sep; }
        if ( $email eq "" )
        {print $sep;}
        else { print $quo . $email . $quo; }
        print "\n";

    } # end if "END:VCARD"
    
} # end while STDIN

Note that I also added a header line for the CSV file, which Gmail seems to expect.

To use the above two scripts, copy and paste each of them into their own file, named exportevolution.py and vcard2csv.pl. Make sure the executable bit is set so that your Linux / BSD / Unix OS will allow you to treat them as executable scripts via:

chmod a+x exportevolution.py vcard2csv.pl

Next, pipe the output from out evolution exporter to the vcard to csv converter, and save it to a suitably named file:

./exportevolution.py | ./vcard2csv.ple > mycontacts.csv

Finally, log into your Gmail account, click on the Contacts link at the upper right hand side of the page, and then import the mycontacts.csv file using the import link Gmail provides. Enjoy!

Anatomy of a 419 Scam

The Register has a great story on the Anatomy of a 419 Scam. The 419 scam is most commonly known as the Nigerian Email scam, which most people have run across in their INBOX at one time or another. The initial contact email usually tells the receiver that they may be in line to inherit X million dollars, as someone has died and left no heirs. The catch, of course, is that you'll need to send them X thousand dollars to establish a bank account into which these funds will (never) be deposited.

This article illustrates a variation on this theme, and comes complete with the emails sent to the mark, as well as his responses. The result is, of course, not good. Furthermore, the article goes on to illustrate why these scams are so popular - because they work, and law enforcement doesn't seem to want to do much about them.

Python Development with Eclipse

As reported here earlier, Eclipse 3.0 was recently released to the computing public. While Eclipse is viewed primarilly as an integrated development environment for the Java programming language, it is actually far more flexible. The Eclipse Foundation has positioned Eclipse as a general purpose IDE, where plugins for each supported language are needed to cater to the language's specific needs. Therefore, the Java IDE is a separate package from Eclipse, etc.

To develop in Python with Eclipse, read through this article, from the folks at IBM developerworks. It discusses how to download and install the Python IDE plugin, and then goes over some of its features. Very cool!

Eclipse 3.0 Released and More Java Fun!

The Eclipse Foundation has released version 3.0 of the Eclipse IDE. This release debuts run-time binary compilations of the IDE for increased performance, as well as all kinds of new features. Download it and be merry!

Once you've downloaded and installed the new version, try your hand at CodeRuler, a graphical simulator designed to exercise your Java programming skills. Download CodeRuler here.

Flash Game Fun

Here are two flash games that you can use to recharge a few brain cells during some quiet time at work. The first is yet another in the genre of Yeti Sports. Check out Yeti Sports Part 5 - Flamingo Drive and give it your best.

If that doesn't get your fingers quaking, then perhaps a bit of frantic 3D pong will do the trick for you. Curve Ball is a fun game which gets more difficult as you progress through the levels. My personal best so far is reaching Level 8, ending with a score of 24,950.

Fun MiniBall Flash Game

MiniBall is a flash game sutiable for one or two players, which mimics the game of foosball in your web browser. It's a good way to kill the last hour of a tough Monday work day, trust me.

Gmail - Google's Email Service

Although today is April Fool's day, this is no joke. Google, every savvy internet user's favorite search engine, is branching out to the email arena. Currently, Gmail is in a limited beta test phase, but may soon expand their test group to include more people. It sound as if they will be integrating spam filtering as well as a few other features like organizing related email as "conversations", and allowing the user to search through all their email. With a one gigabyte data limit, a user wouldn't need to worry about continually purging older emails. Here is another article on Gmail to further whet your appetite.

SCO as Microsoft Shill?

The Inquirer has a great article summarizing the details which have arisen in investigating the connection between SCO and Microsoft. If you haven't been following the goings-on as SCO attempts to hijack the work that hundreds of people have freely put into Linux, then the article will provide you with a very humorous overview of the current situation.

More SpamAssassin Fun

It appears as if we are all stuck with spam for the foreseeable future, so it is now up to us to do something about it. Here at work, I've installed SpamAssassin as a spam filter that works in conjuntion with the rest of our mail delivery software (those being qmail, procmail, and Courier IMAP).

The path an incoming ei-mail takes can be briefly summarized as follows: qmail receives the incoming e-mail arriving from another SMTP server on the internet. qmail then passes the newly arrived e-mail to procmail which is in charge of running the e-mail through SpamAssassin, and then routing the e-mail to the appropriate mail folder (INBOX if it is not flagged as spam, or the Spam folder if it is). Finally, the user retrieves the message using an e-mail client which supports the IMAP protocol (almost all do, although Mozilla Thunderbird is one open source cross-platform e-mail client which I usually recommend to most people).

I have had this setup running for the last 9 months or so, and it has gone a long way towards minimizing the time wasted on weeding through spam. However, the version I had installed on the Debian GNU/Linux mail server was a bit long in the tooth. To install the newest version of SpamAssassin on the Debian stable server, I added a new repository to the /etc/apt/sources.lst file, pointing it to “deb http://www.backports.org/debian/ woody spamassassin”. A simple ‘apt-get update; apt-get upgrade-u’ and a few seconds later the latest version of SpamAssassin was humming along nicely.

The next step was to implement some of the new features which had been added to SpamAssassin since the last version I had installed. It was time to turn the mail server into a merciless spam-terminating machine. SpamAssassin now has built into it a Bayesian filter module which, in essence, can learn to recognize spam and non-spam (called ‘ham’) e-mails, if you train it. All you need to do to train SpamAssassin is to let it look over 200+ spam and 200+ ham messages, and then it is ready to add the Bayesian filter check to its repertoire of tests.

Instead of doing that for each and every user on the system, I decided to go ahead and install the filter on a site-wide basis. This caused a few problems due to permission issues, but I finally hammered out a nice solution that seems to be working well. What is even cooler (in an admittedly geeky sort of way) is that I've created a process by which spam that gets through the varied defenses can be assimilated (very Borg-like, eh?) into the filter, thereby continually fine tuning it through time. Basically, I created a sub-folder of the Spam folder called “Missed”. If a spam message sneaks past the defenses, then I've instructed the users here at work to slap it in the Missed folder. Every day the mail server then executes a shell script which goes through each user's Missed folder, learns from the spam it missed, and then deletes those messages! Here is the shell script, which is executed by the cron scheduler:

#!/bin/sh
                                                                                                                                           
# train-spam.sh
#
# Description: Checks each user's /home/Maildir/.Spam.Missed
# directories to see if the user placed any "missed" spam
# messages which got through SpamAssassin to their INBOX.
# If there are messages in this directory, then the script
# invokes sa-learn to update the site-wide tokens to try
# and improve the defenses for next time...
#
                                                                                                                                           
for file in $(ls /home); do
                                                                                                                                           
    if [ -d /home/$file/Maildir/.Spam.Missed/cur ]; then
                                                                                                                                           
        echo -n "missed spam for $file: "
                                                                                                                                           
        # run sa-learn on the conents of the directory
        sa-learn --spam -C /etc/spamassassin --showdots --dir /home/$file/Maildir/.Spam.Missed/cur
                                                                                                                                           
        # delete all of the missed messages
        rm -f /home/$file/Maildir/.Spam.Missed/cur/*
                                                                                                                                           
        echo "Done!"
                                                                                                                                           
    fi # end if
                                                                                                                                           
    # set up permissions on site-wide token files to allow
    # all users permission to read from and write to these files
    chmod a+r /etc/spamassassin/bayes_*
    chmod ug+w /etc/spamassassin/bayes_*
    chgrp users /etc/spamassassin/bayes_*
                                                                                                                                           
done # end for loop
                                                                                                                                           
echo "Done!"

Hopefully this solution will reduce spam which slips under the radar by 90% or more. Only time will tell...

Great Article on Closed Computing and Future Consequences

Many of my friends and family know that I am a big fan of Free/Open Source Software (FOSS). Many of them look at me as if I've stumbled out of the forest after 40 days of who-knows-what, wearing a tattered loain cloth with bits of locust and wild honey clinging to a bedraggled beard. Some of them will ask me “Why?”, and I do my best to explain the many benefits of FOSS, but by that time most people have that glazed-over look in their eyes, which concludes the discussion.

This article by Doug Dingus does a great job in taking something that most people are familiar with (cellular phone companies and calling plans), and relates it to the dangers of a closed computing. Read it - it's good stuff!

Firebird Reborn as Firefox

My favorite web browser from the good folks at The Mozilla Organization was Firebird. I say ‘was’ because with the expected feature and performance upgrades in the newest release (version 0.8) also comes a change in name - to Firefox. The change in name is due to another open-source project called Firebird (a DBMS based on Borland's Interbase server).

So how does Firefox feel? Much snapier, and everything seems to run smoother than version 0.7. A new download manager is present, and my initial impression is that it is an improvement over the previous d/l manager. So download it and give it a spin.

Not to be out done, the Thunderbird team has released version 0.5. Thunderbird is the e-mail and newsgroup component of the Mozilla product line. Seems to be coming along nicely!

Programming Linux Socets

IBM's DeveloperWorks Tutorials site is a great site to keep your eye on if you are a computer programmer. They cover topics which range from Java to Python, XML to web services, etc. Note that to access the tutorials and other articles on the site, you must first go through a quick (and free!) registration process, which is well worth the effort.

As an example of what you can find there, take a look at this introduction to Programming in Linux Sockets, Part 1. This introduction starts out with the basics of IP networks and network layers, and then proceeds to show you how to applications which utilize these parts in C and Python.

How I PC'd an Apple G5

Several computer-tech related web sites publish articles (or HOW-TOs) detailing how a new motherboard was installed into an old or unusual case, or how a case has been tricked out with a frozen nitrogen cooling system for the CPU so that it can be overclocked to 99.5 GHz. However, This article on how a gent converted his a $3,000 Apple Dual G5 to an Athon 2400 system (less than $1,000) is in a somewhat different vein. You see, it was a hoax. It appears as if some people just can't take a jok... The reaction to the hoax from induhviduals of the Mac community was so quick and brutal that it quite caught the author off guard. Wired also has an article on the whole affair, which I find enormously hillarious.

Mozilla Firebird Tips and Tricks

I use Mozilla Firebird as my web browser for both my Debian Linux and Windows XP boxes at work and home. The browser is clean, slick, effecient, and has lots of nice features like pop-up blocking, tabbed browsing (essential once you are introduced to it), etc.

It turns out that Mozilla Firebird is highly tweakable as well. The Mozilla Firebird: Tips and Tricks page details some of the more esoteric tricks to make Firebird more to your liking. As an example, you can block most ads from appearing in web pages you visit, use the download sidebar instead of a progress window, and more. Check it out if you are a Firebird afficianado!

The OQO Ultra Personal Computer

The OQO uPC seems to be one of the bigger hits at this years COMDEX convention in Las Vegas, Nevada. This little baby is due out in the latter half of 2004 with a sticker price reportedly under $2,000. That is a lot of change for not-too powerful PC, but you are paying for the size (or lack thereof) of the unit. With a built in thumb-style keyboard AND pen pointing device, it looks very interesting. Add in the 802.11b wireless adapter, and you have a truly useful and small Windows XP system.