• MySQL -> PostgreSQL
• ACCESS/MYSQL -> ACCESS/ODBC (MySQL) ->ACCESS/ODBC (pgODBC) -> ACCESS/ODBC (DataDirect 64-bit)
• LATIN1 -> UTF-8
• SAS 9.1.3 -> SAS 9.2

Most of these transitions went pretty straightforward.  However, one bug got introduced somewhere along the way and I just couldn’t ever seem to figure out what would cause it.

For some reason, to download a list of features to get real ID numbers and then match by name, this wouldn’t work:

I’d get bizarre errors out of SAS that the list wasn’t sorted.  Whenever it occurred, I’d inspect the resultant (and intermediate) datasets, and everything seemed sorted just fine.  Instead, I had to have something like this:

You can clearly see my frustration (and the blaming of my employer’s own software over Postgres) in the comments.  In addition, I probably overkilled the solution by re-sorting and then also creating an index, but it did make the problem go away.

Eventually, I got another bug report from a user with the phrasing, “Incorrect sorting within letter group in Features table”.  That led me to this entry in the Postgres wiki:

http://wiki.postgresql.org/wiki/FAQ#How_do_I_change_the_sort_ordering_of_textual_data.3F

I discovered SAS sorts strings based on rules found in “C” locale collation. Even though I read some documentation attributing the default LC_COLLATE setting as “C”, in fact, for my database, it was set to “en_US.UTF-8″. What this basically means is when sorting the following list:

• GLMMOD : Tests
• GLM : ODS Graphics
• GLM : Checklist
• GLMMOD : Checklist

You’ll get:

1. GLM : Checklist
2. GLMMOD : Checklist
3. GLMMOD : Tests
4. GLM : ODS Graphics

Which seems incorrect at first, until you realize it sorts disregarding whitespace and special characters.  However, SAS and “C” locale collation sort like this:

1. GLM : Checklist
2. GLM : ODS Graphics
3. GLMMOD : Checklist
4. GLMMOD : Tests

Because PROC SORT and other SAS mechanisms issue and rely on native database commands for some operations, this behavior can produce results in ordering undesirable for SAS.  SAS actually performed very admirably by delegating the sorting to the database, setting the appropriate flags on the dataset, but then still catch the match merge problem at runtime!

Long story short, when using PostgreSQL with SAS, it’s probably a good idea to make sure the database is created with the correct setting for LC_COLLATE. If it is not, you may end up with crazy gyrations like mine in your code.  Luckily for me, it’s a fixable scenario whereby the database only needs dumped, then restored after it has been recreated with the desired collation.

I eventually stumbled on the Mac’s console message viewer.  He’s extremely handy.  He zeroed me in on the culprit, “Incorrect number of thread records (#, #)”.  Okay, great message.  Maybe if you know the number is incorrect, you could tell me the correct number?  Maybe if you know the number is incorrect, you could just fix it?  Nah.

After some more research, the next remediation steps were to boot into the install disk, try running Disk Utility and also to start up a console, unmount the main volume and run fsck_hfs manually.  Nada.  All of those seemed to know all about my incorrect thread count, but none could make a correction.  I must admit though, with a lot of Windows (and Linux) diagnostics behind me, the utilities are at least navigable well presented on the install disk.

With the easy stuff behind us, my options now get expensive or drastic.   Most of the forum and support threads I found have DiskWarrior as the next logical step to try and fix the volume.  At $100, I’m not quite ready to give up.  I’m close though because with a BootCamp volume and a pretty important Windows 7 install on the other partition, formatting  and redoing all that work doesn’t sound palatable… yet.

I do have AppleCare.  I use the online form, open a ticket and schedule a callback on 12/31.  Cool!  The phone rings.  The automated attendant informs me they are closed.  What!?  Why can I schedule the call then?  The call ends.  Two minutes later, I’m called again by Mr. Robot.  Ugh.  I re-schedule for another day to avoid infinite recursion!  A slick customer service interface has a bug in it lets me schedule calls when they are on holiday and then has another bug in you can get in a loop.

OCD kicks in.  I want this to go away now.  What do I have unique to the Windows partition, really?  Just a Quickbooks file, that’s not a big deal, back it up separate.  Time Machine backup is a little behind, let’s take a new one.  Okay, I’ll use the install disk to erase the Mac drive and try re-installing Snow Leopard.  I’m sure my users and data will restore from Time Machine.  I may have to do some homework to access my Windows partition (if it survives).  I’ll probably have to redo my network setup, install iLife, Firefox, eclipse, etc.  If my Windows partition doesn’t survive, that’ll be fun the next few days…

The install is pretty quick, 30-60 minutes.  No problem.  It restarts.  Here comes my getting started page.  There’s a migrating/moving via Time Machine button.  Sure.  The next screen was pretty surprising.  I had options for restoring four things… something like users, data, settings and apps!  Are you serious!?  Wow.  Something will probably not be quite right, but this is HUGE and a big divergence from my non-Mac experiences…

The Time Machine restore cranks through my files.  It completes.  I restart.

At the gray startup screen, I hold down “Alt”.  Will Windows still be there?  It is!  Yay!  He comes up just fine.  That, in and of itself, is extremely cool.  I had a feeling the partition would be there just from the screens I was seeing on the OS X install disk.  Not sure about Vista or 7, but Windows prior would always blow away any bootloader I would have, no question.

I then start OS X.  Surprise, surprise, I must have the correct number of thread records now, it boots in normal time .  The login screen has my users, their little thumbnails, cool.

Logging on was what really blew my mind.  Once inside, all updates were already applied, the network settings are all there and everything/EVERYTHING was there; iLife (a separate install), an Eclipse I had downloaded and extracted somewhere funny, every other app and setting.  Wow.  I was so dumbfounded and filled with confidence about the whole process if it takes me more than an hour to find a button to undo a setting not 100% to my liking in the future, I will re-install from scratch and use the last backup not containing it in a heartbeat…  For the Mac-savvy, or anybody who read into Time Machine since it came out, this may seem funny, but I still can’t get over it.

It didn’t escape me, going through the motions, OS X was treating it the same as if I had purchased a new computer and was just “moving” to it with my Time Machine backup.  Companies don’t use Linux desktops (yet, ugh), but is it this smooth for the minority that do when going to new hardware?  I doubt it.  For companies using Windows desktops, entire armies of IT guys are still trying to purchase or develop applications/processes/scripts to make moving to new hardware this easy for the employees at large when hardware refreshes are in order.  I know it wasn’t this smooth for me when I got my new gear the last time (XP -> XP).  I have another Windows upgrade coming in January (XP -> 7), we’ll see how that goes…

Ad-Hoc Burning

growisofs -Z /dev/dvd -R -J /path/to/files

- later -

growisofs -M /dev/dvd -R -J /more/files

- to finalize -

growisofs -M /dev/dvd=/dev/zero

Writing an ISO

growisofs -dvd-compat -Z /dev/dvd=/path/image.iso

Erase the Disk

growisofs -Z /dev/dvd=/dev/zero

Format

dvd+rw-format -ssa=1G /dev/dvd

I’ve recently been trying to do some new things and I thought I would post that as of Fedora 11, Brasero can recognize and write files to my BD-RE media and also erase the disk to do it again.  K3b is still at 1.0.5 (not a KDE 4 compatible version) on Fedora and does not recognize the disk correctly for type and capacity, nor does it allow burning.

My recent searches pull up the same results from 2007 and 2008 as before where people were unsure, etc.  This is the current state of my world though.

*UPDATE 2010-06-25*

It’s been awhile since I’ve tried this, but now using Fedora 13, I am able to use K3B to burn single-layer BD-RE media at 2.0x speed.  I have a Sony BDRW BWU-200S.

Pleas see this attachment for the source code described in this article.

The method I will outline here requires the php-soap module.

First, the WSDL.  For my project, I started with a WSDL created in another tool.  My WSDL specifies a slightly different object set than my CakePHP application.  I’m sure with PHP5 and some finessing of the Model classes, you could probably use the same set; however, it was easy enough to just create some really vanilla objects to house the transport objects and use those to communicate with the webservice.  Both the WSDL and the receiving controller are present in the attachment.

What you will notice is that the *DTO objects defined in the controller file reciprocate the structure of the objects in the WSDL and the methods also are represented in the controller.  I put them in the controller file because it wasn’t really obvious to me where in Cake’s structure “outside code” should really go.  I have a separate configs.inc.php I pull in up the class hierarchy, but that’s about as non-conventional as I want to get.  Also, this controller is dedicated to just handling webservice requests and I only need these *DTO objects in that case, so locality wins and they are here.  No real engineering genius here, their structure mimics what is defined in the WSDL file.

The real magic is in the controller.  The controller’s remote() method is what handles the POST from the web via the port binding in the WSDL file.  The remote() function sets up some of the basic stuff for SoapServer and is easily identified in the PHP manual.  It’s even pretty easy to deduce we’re going to need to use SoapServer->setClass() somewhere and plug the name of our Controller in.   However, there was one tidbit in the comments section of the manual regarding SoapServer->setObject().  It wasn’t documented (at the time), but after experimenting and looking at the PHP source, it does exactly what we need here, sets the handling class to an instantiated (aka existing) class object instead of trying to spawn a new one.  Because we are already inside the CakePHP framework and running the remote() function, we already have the variables we want from beforeFilter(), we have our models loaded up, we may even have a user context from mod_auth_something.  Perfect!!!  So, we tell SoapServer to use our instantiated Controller.  Once the *DTO classes are mapped and SoapServer is configured, it’s as simple as having it handle STDIN to tickle the rest of the methods in your Controller with the parameters populated.  Two more tricks/problems remain:  debug level & autoRender.

First, debug level.  There’s bound to be a way around it; however, since I test with a web service client, when I do have a problem, I have to debug with lots of $this->log() calls.  Turning up debugging to 1 or 2 is problematic because then CakePHP doesn’t spit back properly formed XML to the web service client and usually the client takes a SoapFault when that happens.  I stick to debug level of 0 during development and deployment wrt the web service stuff.

Second, autoRender.  Because SoapServer does the actual outputting of XML response to the client, I set the layout in the Controller to Ajax and also explicitly call exit() at the end of the remote() method.  This ensures that CakePHP doesn’t send back a “Missing View”, half rendered $layout, or any other kind of automatic goodies.

I hope this article is helpful for anybody who might want/need to integrate a more elegant/esoteric webservice into their CakePHP architectures.  I’m sure there are probably cleaner ways to put this into custom View classes, utilize Components, etc… however, this was a straightforward approach I found has been working really well for one of my applications.

Initial creation of the type and table:

CREATE TYPE var_type AS ENUM('text', 'number', 'date', 'boolean');

CREATE TABLE custom_fields (
    id bigserial PRIMARY KEY,
    name varchar(50) NOT NULL,
    pdf_type var_type NOT NULL
);

Running with this table for some time, invariably, new rows are created and there’s now a migration consideration.  As long as you are not using the table column as a reference in a foreign key, the following should work to preserve the data, drop and re-create the type.

The following creates a new column to hold the original text value:

ALTER TABLE custom_fields ADD COLUMN type_text varchar(15);
UPDATE custom_fields SET type_text = pdf_type::text;

We, then, need to drop the existing type and re-create it with the new values we want.  CASCADE automatically drops columns that depend on the type:

DROP TYPE var_type CASCADE;
CREATE TYPE var_type AS ENUM('text', 'number', 'date', 'boolean', 'list');

This last part was what I couldn’t figure out without thinking a little more.  When you add it back, you have to cast the varchar column back into the ENUM type.  I had tried a variety of concoctions here before getting this to work:

ALTER TABLE custom_fields ADD COLUMN pdf_type var_type;
UPDATE custom_fields SET pdf_type = type_text::var_type;
ALTER TABLE custom_fields ALTER pdf_type SET NOT NULL;
ALTER TABLE custom_fields DROP COLUMN type_text;

So here are the few lines of Perl that got the job done:

#!/usr/bin/perl
if ($#ARGV + 1 < 1) {
        print "\n\tUsage:\n";
        print "\t------\n\n";
        print "\tperl log_anonymize.pl file1 [file2 [file3 [...]]]\n\n";
        die "Please specify at least one file to use this script.\n\n";
}

my %forward = ();
my %reverse = ();

foreach (@ARGV) {
        open(ORIG, $_)
          or die "Failed to open input file for reading.";
        open(ANON, "+>", $_.".anon")
          or die "Failed to open destination file for writing.";
        while (<ORIG>) {
                if (/([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)/) {
                        if (!($forward->{$1})) {
                                $newIp = getNewIp();
                                while ($reverse->{$newIp}) {
                                        $newIp = getNewIp();
                                }
                                print "New mapping created: $1 -> $newIp\n";
                                $forward->{$1} = $newIp;
                                $reverse->{$newIp} = $1;
                        }
                        $repl = $forward->{$1};
                        $_ =~ s/$1/$repl/;
                }
                print ANON $_;
        }
        close(ORIG);
        close(ANON);
}

exit 0;

sub getNewIp {
        return int(rand(256)) . "." . int(rand(256)) . "." . int(rand(256)) . "." . int(rand(256));
}

It is fairly straightforward.  You invoke the Perl script with one or more arguments.  Every argument should be a path to an access log.  For each file, a new file of the same name and “.anon” appended gets created.  Across all those files, the script maintains an internal hash of the IPs it has mapped to a new, random IP address and will re-use those mappings as they are encountered.  It spits out a little message when the mappings occur so you could do some counts using ‘wc’ or something similar to see how many you had… or you could make it output a count at the end, it’s pretty simple to do either.

So that’s it, easy web log anonymizing via random IP address remapping.

SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration.