• MySQL -> PostgreSQL
• ACCESS/MYSQL -> ACCESS/ODBC (MySQL) ->ACCESS/ODBC (pgODBC) -> ACCESS/ODBC (DataDirect 64-bit)
• LATIN1 -> UTF-8
• SAS 9.1.3 -> SAS 9.2

Most of these transitions went pretty straightforward.  However, one bug got introduced somewhere along the way and I just couldn’t ever seem to figure out what would cause it.

For some reason, to download a list of features to get real ID numbers and then match by name, this wouldn’t work:

I’d get bizarre errors out of SAS that the list wasn’t sorted.  Whenever it occurred, I’d inspect the resultant (and intermediate) datasets, and everything seemed sorted just fine.  Instead, I had to have something like this:

You can clearly see my frustration (and the blaming of my employer’s own software over Postgres) in the comments.  In addition, I probably overkilled the solution by re-sorting and then also creating an index, but it did make the problem go away.

Eventually, I got another bug report from a user with the phrasing, “Incorrect sorting within letter group in Features table”.  That led me to this entry in the Postgres wiki:

http://wiki.postgresql.org/wiki/FAQ#How_do_I_change_the_sort_ordering_of_textual_data.3F

I discovered SAS sorts strings based on rules found in “C” locale collation. Even though I read some documentation attributing the default LC_COLLATE setting as “C”, in fact, for my database, it was set to “en_US.UTF-8″. What this basically means is when sorting the following list:

• GLMMOD : Tests
• GLM : ODS Graphics
• GLM : Checklist
• GLMMOD : Checklist

You’ll get:

1. GLM : Checklist
2. GLMMOD : Checklist
3. GLMMOD : Tests
4. GLM : ODS Graphics

Which seems incorrect at first, until you realize it sorts disregarding whitespace and special characters.  However, SAS and “C” locale collation sort like this:

1. GLM : Checklist
2. GLM : ODS Graphics
3. GLMMOD : Checklist
4. GLMMOD : Tests

Because PROC SORT and other SAS mechanisms issue and rely on native database commands for some operations, this behavior can produce results in ordering undesirable for SAS.  SAS actually performed very admirably by delegating the sorting to the database, setting the appropriate flags on the dataset, but then still catch the match merge problem at runtime!

Long story short, when using PostgreSQL with SAS, it’s probably a good idea to make sure the database is created with the correct setting for LC_COLLATE. If it is not, you may end up with crazy gyrations like mine in your code.  Luckily for me, it’s a fixable scenario whereby the database only needs dumped, then restored after it has been recreated with the desired collation.

Pleas see this attachment for the source code described in this article.

The method I will outline here requires the php-soap module.

First, the WSDL.  For my project, I started with a WSDL created in another tool.  My WSDL specifies a slightly different object set than my CakePHP application.  I’m sure with PHP5 and some finessing of the Model classes, you could probably use the same set; however, it was easy enough to just create some really vanilla objects to house the transport objects and use those to communicate with the webservice.  Both the WSDL and the receiving controller are present in the attachment.

What you will notice is that the *DTO objects defined in the controller file reciprocate the structure of the objects in the WSDL and the methods also are represented in the controller.  I put them in the controller file because it wasn’t really obvious to me where in Cake’s structure “outside code” should really go.  I have a separate configs.inc.php I pull in up the class hierarchy, but that’s about as non-conventional as I want to get.  Also, this controller is dedicated to just handling webservice requests and I only need these *DTO objects in that case, so locality wins and they are here.  No real engineering genius here, their structure mimics what is defined in the WSDL file.

The real magic is in the controller.  The controller’s remote() method is what handles the POST from the web via the port binding in the WSDL file.  The remote() function sets up some of the basic stuff for SoapServer and is easily identified in the PHP manual.  It’s even pretty easy to deduce we’re going to need to use SoapServer->setClass() somewhere and plug the name of our Controller in.   However, there was one tidbit in the comments section of the manual regarding SoapServer->setObject().  It wasn’t documented (at the time), but after experimenting and looking at the PHP source, it does exactly what we need here, sets the handling class to an instantiated (aka existing) class object instead of trying to spawn a new one.  Because we are already inside the CakePHP framework and running the remote() function, we already have the variables we want from beforeFilter(), we have our models loaded up, we may even have a user context from mod_auth_something.  Perfect!!!  So, we tell SoapServer to use our instantiated Controller.  Once the *DTO classes are mapped and SoapServer is configured, it’s as simple as having it handle STDIN to tickle the rest of the methods in your Controller with the parameters populated.  Two more tricks/problems remain:  debug level & autoRender.

First, debug level.  There’s bound to be a way around it; however, since I test with a web service client, when I do have a problem, I have to debug with lots of $this->log() calls.  Turning up debugging to 1 or 2 is problematic because then CakePHP doesn’t spit back properly formed XML to the web service client and usually the client takes a SoapFault when that happens.  I stick to debug level of 0 during development and deployment wrt the web service stuff.

Second, autoRender.  Because SoapServer does the actual outputting of XML response to the client, I set the layout in the Controller to Ajax and also explicitly call exit() at the end of the remote() method.  This ensures that CakePHP doesn’t send back a “Missing View”, half rendered $layout, or any other kind of automatic goodies.

I hope this article is helpful for anybody who might want/need to integrate a more elegant/esoteric webservice into their CakePHP architectures.  I’m sure there are probably cleaner ways to put this into custom View classes, utilize Components, etc… however, this was a straightforward approach I found has been working really well for one of my applications.

Initial creation of the type and table:

CREATE TYPE var_type AS ENUM('text', 'number', 'date', 'boolean');

CREATE TABLE custom_fields (
    id bigserial PRIMARY KEY,
    name varchar(50) NOT NULL,
    pdf_type var_type NOT NULL
);

Running with this table for some time, invariably, new rows are created and there’s now a migration consideration.  As long as you are not using the table column as a reference in a foreign key, the following should work to preserve the data, drop and re-create the type.

The following creates a new column to hold the original text value:

ALTER TABLE custom_fields ADD COLUMN type_text varchar(15);
UPDATE custom_fields SET type_text = pdf_type::text;

We, then, need to drop the existing type and re-create it with the new values we want.  CASCADE automatically drops columns that depend on the type:

DROP TYPE var_type CASCADE;
CREATE TYPE var_type AS ENUM('text', 'number', 'date', 'boolean', 'list');

This last part was what I couldn’t figure out without thinking a little more.  When you add it back, you have to cast the varchar column back into the ENUM type.  I had tried a variety of concoctions here before getting this to work:

ALTER TABLE custom_fields ADD COLUMN pdf_type var_type;
UPDATE custom_fields SET pdf_type = type_text::var_type;
ALTER TABLE custom_fields ALTER pdf_type SET NOT NULL;
ALTER TABLE custom_fields DROP COLUMN type_text;

So here are the few lines of Perl that got the job done:

#!/usr/bin/perl
if ($#ARGV + 1 < 1) {
        print "\n\tUsage:\n";
        print "\t------\n\n";
        print "\tperl log_anonymize.pl file1 [file2 [file3 [...]]]\n\n";
        die "Please specify at least one file to use this script.\n\n";
}

my %forward = ();
my %reverse = ();

foreach (@ARGV) {
        open(ORIG, $_)
          or die "Failed to open input file for reading.";
        open(ANON, "+>", $_.".anon")
          or die "Failed to open destination file for writing.";
        while (<ORIG>) {
                if (/([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)/) {
                        if (!($forward->{$1})) {
                                $newIp = getNewIp();
                                while ($reverse->{$newIp}) {
                                        $newIp = getNewIp();
                                }
                                print "New mapping created: $1 -> $newIp\n";
                                $forward->{$1} = $newIp;
                                $reverse->{$newIp} = $1;
                        }
                        $repl = $forward->{$1};
                        $_ =~ s/$1/$repl/;
                }
                print ANON $_;
        }
        close(ORIG);
        close(ANON);
}

exit 0;

sub getNewIp {
        return int(rand(256)) . "." . int(rand(256)) . "." . int(rand(256)) . "." . int(rand(256));
}

It is fairly straightforward.  You invoke the Perl script with one or more arguments.  Every argument should be a path to an access log.  For each file, a new file of the same name and “.anon” appended gets created.  Across all those files, the script maintains an internal hash of the IPs it has mapped to a new, random IP address and will re-use those mappings as they are encountered.  It spits out a little message when the mappings occur so you could do some counts using ‘wc’ or something similar to see how many you had… or you could make it output a count at the end, it’s pretty simple to do either.

So that’s it, easy web log anonymizing via random IP address remapping.

SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration.

SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration.

http://alphaworks.ibm.com/tech/namma

In addition, I tend to try and use whatismyip.com in conjunction with this tool (the available, already written plugins need a public web-site that reports the IP address it sees); however, it tends to limit how many times a day you can check. I have created a simple page that lets you get that information:

http://www.cuppett.com/ip.php

Enjoy!