Harry Gates’ blog

Essbase_Perl.pm LibXML update

Fri, 27 Mar 2009 09:54:47 -0400

I have significantly rewritten the subroutine in Essbase_Perl.pm that reads the XML configuration file. It now uses the XML::LibXML Perl module instead of XML::Simple. I made this design decision for 2 reasons.

First, XML::Simple uses internal Perl data storage mechanisms. Those with even the slightest acquaintance with Perl hashes and arrays know of their power and simplicity. However, they suffer from one major flaw when used in conjunction with XML. They do not retain the original order of XML nodes. This is why the original config_load.xml had the rather klunky num="1" numbering system for each command. Without explicitly specifying the order of commands in the xml configuration file, XML::Simple would have run them in whatever seemingly-random order they were stored by Perl's hashes and arrays. Obviously this could be a problem. After all, you wouldn't want to calc your database before even loading it, now would you?

LibXML is written in C, so it is immune from this particular peculiarity of Perl's processing <wipe spit off monitor here if reading aloud>. The order of commands you specify in config_load.xml is now the order in which the commands will be executed. No kludgy command numbering system is required.

Getting rid of the command numbering is a MAJOR advantage, as I soon found out when I had to add a command #35 in the middle of an xml section containing 100 commands. With the old XML::Simple version of Essbase_Perl.pm I had to manually renumber the commands after the new command #35. That meant changing the old num="35" to num="36", the old num="36" to num="37", the old num="38" to num="39"... I think you get the point. Not a lot of fun. With the LibXML version, however, I simply inserted the new command where it needed to go and that was that. No muss, no fuss.

The second reason I made the decision to use LibXML instead of XML::Simple can be summed up in one word: speed. Because it's implemented in C, LibXML is lightning fast. "Well", you might be thinking, "my xml configuration file is only a couple hundred lines long - at most, so we're only talking nanoseconds of difference here"; and you'd be right.

The speed factor really only comes into play when you're reading and writing very large xml files - with tens or hundreds of thousands of lines. And this is exactly the situation I found myself in with one client recently. I needed to read a huge xml file (we're talking several gigabytes in size here) exported from a certain ornery GL, which shall remain nameless (<cough>PeopleSoft</cough>). XML::Simple was choking on the file. LibXML, on the other hand, processed the whole file in 5 minutes. Needless to say, I am now a LibXML convert.

Due to other improvements of using LibXML, the structure of the "General" section of config_load.xml has also been changed. Here's a sample of what it now looks like:

<General>
<desc>use this section to set general variables for use throughout the load</desc>
<variable name="EssUser">harry</variable>
<variable name="EssPwd">rules</variable>
</General>

Previously it looked like:
<General>
<desc>use this section to set general variables for use throughout the load</desc>
<EssUser>harry</EssUser>
<EssPwd>rules</EssPwd>
</General>

I prefer the new way (using "variable"). It makes it crystal clear that we're setting user-defined variables here, not simply populating some kind of predefined, Essbase_Perl.pm options.

So, without further eloquence, here is the modified Essbase_Perl.pm code:
use XML::LibXML;
sub readXMLConfig {
my ($xml) = @_;

my ($return, @msgs, $start, $sec_start);

$parser = XML::LibXML->new();

# Check for well-formedness, stop if issue:
eval {$parser->parse_file($xml);};
&errHandler(1, "issue with $xml config file", $@) if ($@);

$doc = $parser->parse_file($xml);

print "following is the $xml config file at the time of the load:\n";
print $doc->toString;
#print Dumper($doc) . "\n";

my $date = scalar localtime(time);
my $start_load = time;

print "\n\nstarting load process at $date\n";

#set variables from General section of config_load.xml file
foreach my $general ($doc->findnodes('//General/variable')) {
$vars{$general->getAttribute('name')} = $general->textContent;
}

my $dbh = &Ess_connect($vars{EssUser}, $vars{EssPwd});

foreach my $order ($doc->findnodes('//Run_Order/command')) {
my $sec = "//" . $order->textContent;
my $app = $doc->findvalue($sec . "/application");
my $db = $doc->findvalue($sec . "/database");
my $section = $sec . "/command";
my $sec_start = time;
print "\n##################Section is: " . $order->textContent . "#######################################\n";
my $i = 1;

foreach my $config ($doc->findnodes($section)) {
print "\nrunning command #" . $i . "\n";
my $type = $config->getAttribute('type');

&runCommand($type, $config, $app, $db, $dbh);
$i++;
}
my $sec_elapsed = &Elapsed($sec_start, time);
print "$section completed in $sec_elapsed\n";
print "####################$sec completed successfully############################################\n";
&sendMsg($err_to, $vars{notification_from}, "completed $sec", "completed $sec successfully in $sec_elapsed", "");
}
Ess_disconnect($dbh);
print "Successfully completed. Processing took: " . &Elapsed($start_load, time) . "\n";
&sendMsg($vars{notification_to}, $vars{notification_from}, "Successfully completed Essbase processing in: " . &Elapsed($start_load, time), "", "");
#&notify;
}

Update

Wed, 21 Jan 2009 19:39:48 -0500

I haven’t had much time lately to work on the Essbase_Perl.pm module. I started a new job at a consulting company recently and have been extremely busy. Most of the work I will be doing for the next couple of months is Excel VBA-related. I have, however, done several installs of Essbase 11.1.1 - both on Windows and Linux - for clients.

The first step, of course, is to download the Linux version of EPM 11.1 from Oracle’s website.

The new installer works the same on both platforms, but there are several things to watch out for on Linux. Hopefully when I get more time, I’ll be able to post some more details.

I was unable to get the EAS (Essbase Administration Server), the program, not the service, to run on Linux. I can’t tell from Oracle’s documentation whether EAS is supported on Linux. Does anyone know?

Binaries of Essbase.pm compiled for win32

Wed, 3 Sep 2008 11:00:32 -0400

There seems to be great demand for binaries of the Essbase.pm compiled for Windows. Therefore, I am making mine available. I am using ActivePerl 5.10.0 for MSWin32-x86. The binaries below work for me on Windows XP SP2 and Windows 2000 SP4. I used Microsoft Visual C++ 6.0 to compile them.

If you have installed ActivePerl with the defaults, Perl should be located under C:\Perl. Given that, here is where the files need to go:

under C:\Perl\site\lib
essbase.pm
under C:\Perl\site\lib\auto\Essbase
Essbase.dll
Essbase.exp
Essbase.bs
Essbase.lib

That’s it. It’s really not that hard. And it’s definitely worth the effort, because once you have the Essbase Perl module installed and working, you can start using my Essbase_Perl.pm to make your life much easier.

Essbase_Perl.pm subroutines (sending email notifications and disconnecting)

Wed, 27 Aug 2008 11:05:02 -0400

Previously we reviewed how the Essbase_Perl.pm handles errors. It’s not enough, however, just to identify an error and stop the program from proceeding. Wouldn’t it be great to have the program send an email notifying a designated person (and let’s face it, it’s us, the lowly Essbase admin) to take a look at the issue?

As most of the Essbase loads I’ve been involved with are set to run overnight, this facility is a godsend. No longer do I have to sit up all night babysitting my Essbase loads. No, sir. Now I can sleep like a baby, knowing that any errors will be sent to my trusty BlackBerry, which has been preset to emit an ear-piercing shriek whenever it receives an error email from Essbase_Perl. Enough talking. Let’s dive into the code.

sub sendMsg {
my ($to, $from, $subject, $message, $file) = @_;

my $msg = MIME::Lite->new(
From => $from,
To => $to,
Subject => $subject,
Data => $message
);

if (defined ($file) and ($file ne "")) {
my @files = split(/,/, $file);

foreach my $f (@files) {
$msg->attach(
Type => "text/plain",
Path => $file,
Filename => $file,
Disposition => "attachment"
);
print "attached file: $file\n";
}
}

MIME::Lite->send('smtp', 'enter_name_of_mail_server_here', Timeout => 60);
my $ret = $msg->send();
return 0 if $ret == 1;
}

In order for the subroutine above to work, you will need to add a reference to the MIME Lite module in your Perl file, like so:

use MIME::Lite;

Clear as mud? Let’s see if I can make a little sense out of it for you. The first line is getting the parameters. As you can see, the ‘to’ email address, the ‘from’ address, the ‘subject’, body, and any files to be attached are passed in by the calling subroutine.

So what, Harry, does the following line do?

if (defined ($file) and ($file ne "")) {

Sometimes you don’t want, or need, to attach files to the email message. In this case the $file variable will be empty. If that’s the case, there’s no reason to run the code to attach each file. If, however, you would like to attach log files, for example, containing the errors, you can pass them as a comma-delimited string into the $file variable.

That’s where the @files array comes into play. It takes the comma-delimited string and splits it into an array. The next line, beginning with ‘foreach’, loops through each value in the array, representing a filename, and attaches it to the email. The last few lines send the email.

One thing to remember, however. is to change ‘enter_name_of_mail_server_here’ to whatever your company’s mail server is. An example would be ‘mail.mycompany.server’. Basically, it can be just about anything that your email administrator can dream up. It’s best just to find him and ask.

Finally, after running your program (with no errors, I’m sure), you’ll want to disconnect from Essbase. Otherwise, you could have some orphaned processes floating around on your Essbase server. And we can’t have that, can we? Thankfully, the Essbase.pm module from Hyperion provides a function specifically for disconnecting.

sub Ess_disconnect {
my ($dbh) = @_;
print $dbh->disconnect() == $Essbase::MAXL_STATUS{NOERR} ?
"ok: Disconnect from Essbase\n" :
"not ok: Disconnect from Essbase\n";
}

This one’s pretty self-explanatory. It calls the disconnect() function and prints the return value.

In my next post, I’ll discuss a couple of subroutines to run Alter and Aggregate MaxL commands.

Essbase_Perl.pm subroutines (handling errors)

Tue, 26 Aug 2008 16:47:26 -0400

In the last post I discussed the exeMAXL subroutine of the Essbase_Perl.pm. You’ll recall that this subroutine is used to run MaxL commands, return the resulting messages, and check for any errors. But what happens if running the command does result in errors? Well, that’s the topic of today’s post.

If you remember from yesterday’s post, the exeMAXL subroutine had the following line of code after running the MaxL command:

&errHandler($dbh, $cmd, @messages) if $result != 0;

If the return code ($result) from the MaxL command is not equal to zero (zero meaning that no errors occurred), the errHandler subroutine is called from exeMAXL. Here’s what it looks like:

sub errHandler {
my ($dbh, $cmd, @errors) = @_;
my $ret = &sendMsg($err_to, $err_from, "error occurred in $cmd", join("",@errors),"");
print "***********stopping load process due to following error in $cmd:\n@errors";
&Ess_disconnect($dbh) if $dbh != 1;
exit;
}

In the first line, the database handle ($dbh) is passed in, along with the MaxL command and the messages (including the error message) resulting from the command’s execution. The second line calls the sendMsg subroutine, which uses a Perl library (Mime::Lite) to send an email informing you of the error and including the command and error messages. The next line prints the same information. Next, we call a subroutine to disconnect from the Essbase server. The final line exits the code, preventing any further MaxL commands from being executed.

We’ve briefly encountered two new subroutines (sendMsg and Ess_disconnect). In my next post I’ll discuss these in more detail.

Essbase_Perl.pm subroutines (executing maxl commands)

Mon, 25 Aug 2008 20:14:01 -0400

When we left off yesterday I promised to discuss the Essbase_Perl.pm in more detail. Let’s jump right in.

The heart of this package is the exeMAXL subroutine. As the name suggests, this sub executes a MaxL command. Taking it line by line, the first line -- my ($dbh, $cmd) = @_; -- gets the arguments passed into the sub. $dbh represents a database connection and is returned from the second sub below -- Ess_connect. The second argument -- $cmd -- is the actual MaxL command to be executed.

The second line, containing $dbh->do, actually runs the MaxL command, returning the exit code to $result. If the command ran without errors, $result should be equal to zero. The next line checks $result and prints the result.

The fourth line, beginning with ‘my @messages’, returns the messages from the Essbase stack from running the command and stores them in the @messages array.

The next few lines call an error handling routine (errHandler) if any errors occurred. I’ll discuss this sub in detail tomorrow.

The last line returns the result and the messages to the calling subroutine.

sub exeMAXL {
my ($dbh, $cmd) = @_;
my $result = $dbh->do($cmd);

$result == 0 ? print "ok: $cmd\n" : print "not ok: $cmd\n";

my @messages = &msgs($dbh);
$result = 0 if $messages[1] =~ /$non_error/;
&errHandler($dbh, $cmd, @messages) if $result != 0;
return ($result, @messages)
}

sub Ess_connect {
my ($user, $password) = @_;
my $dbh = Essbase->connect($user, $password);

# make sure the database connection succeeded
$dbh->{STATUS} == $Essbase::MAXL_STATUS{NOERR} ? return(0,$dbh) :
return(1,$dbh);

}

So please come back tomorrow for further explanation of Essbase_Perl.pm, my package encapsulating and simplifying the functionality of the Essbase Perl module.

Essbase load program unveiled

Sun, 24 Aug 2008 19:25:54 -0400

I wanted to share with everyone a Perl program that I wrote. This program takes advantage of the power and flexibility provided by the Essbase Perl module (essbase.pm) included in every Essbase installation. You can find it under the {arborpath}\perlmod directory.

WIth the Essbase_Perl.pm you can execute MaxL statements directly from Perl. It then becomes possible to take advantage of the error handling and logic provided by Perl to create a very flexible program for loading data into Essbase databases.

The Essbase_Perl.pm, which I’ll explain in detail tomorrow, is a Perl package that I created as a wrapper for the Essbase Perl module. Because it's not necessary to know the actual MaxL syntax involved, you can accomplish complex tasks by using simple Perl subroutine calls like the following:

my ($return, @msgs) = Essbase_Perl::Ess_import($dbh, "FISDPL", "Main", "D:/Hyperion/AnalyticServices/app/FISDPL_P/Main/xptcrpHIST01", "Load_CY_ASO.xptcrpHIST01", "xptcrp0c", "data", 6, ".flp");

On the other hand, you can configure an XML file (please see config_load.xmll) with the commands, and the order in which you want to run them. The above is represented in the XML configuration file by the following:

<command num="1" type="import" action="data" rules_file="xptcrp0c" num_files="6" file_ext=".flp">
          <data_file>D:/Hyperion/AnalyticServices/app/FISDPL_P/Main/xptcrpHIST01</data_file>
          <error_file>Load_CY_ASO.xptcrpHIST01</error_file>
</command>

This replaces the following lengthy syntax:
MAXL> import database "'FISDPL'"."'Main'" data
   2>    from local data_file      "'D:\Hyperion\AnalyticServices/app/FISDPL_P/Main/xptcrpHIST01.1.flp'"
   3>    using server rules_file   "'xptcrp0c'"
   4>    on error write to         "'D:\Hyperion\AnalyticServices/client/FISDPL/Main/Load_Data_CY.X.xptcrpHIST01.1.err'";

MAXL> import database "'FISDPL'"."'Main'" data
   2>    from local data_file      "'D:\Hyperion\AnalyticServices/app/FISDPL_P/Main/xptcrpHIST01.2.flp'"
   3>    using server rules_file   "'xptcrp0c'"
   4>    on error write to         "'D:\Hyperion\AnalyticServices/client/FISDPL/Main/Load_Data_CY.X.xptcrpHIST01.2.err'";

MAXL> import database "'FISDPL'"."'Main'" data
   2>    from local data_file      "'D:\Hyperion\AnalyticServices/app/FISDPL_P/Main/xptcrpHIST01.3.flp'"
   3>    using server rules_file   "'xptcrp0c'"
   4>    on error write to         "'D:\Hyperion\AnalyticServices/client/FISDPL/Main/Load_Data_CY.X.xptcrpHIST01.3.err'";

MAXL> import database "'FISDPL'"."'Main'" data
   2>    from local data_file      "'D:\Hyperion\AnalyticServices/app/FISDPL_P/Main/xptcrpHIST01.4.flp'"
   3>    using server rules_file   "'xptcrp0c'"
   4>    on error write to         "'D:\Hyperion\AnalyticServices/client/FISDPL/Main/Load_Data_CY.X.xptcrpHIST01.4.err'";

MAXL> import database "'FISDPL'"."'Main'" data
   2>    from local data_file      "'D:\Hyperion\AnalyticServices/app/FISDPL_P/Main/xptcrpHIST01.5.flp'"
   3>    using server rules_file   "'xptcrp0c'"
   4>    on error write to         "'D:\Hyperion\AnalyticServices/client/FISDPL/Main/Load_Data_CY.X.xptcrpHIST01.5.err'";

MAXL> import database "'FISDPL'"."'Main'" data
   2>    from local data_file      "'D:\Hyperion\AnalyticServices/app/FISDPL_P/Main/xptcrpHIST01.6.flp'"
   3>    using server rules_file   "'xptcrp0c'"
   4>    on error write to         "'D:\Hyperion\AnalyticServices/client/FISDPL/Main/Load_Data_CY.X.xptcrpHIST01.6.err'";

The Perl code will run the commands listed in the order specified in the config_load.xml, checking for errors after each command. It is configured to halt the load and send an email with details of the encountered error to an email address also indicated in the config_load.xml configuration file. If no errors are encountered, the program will send an email notifying of its successful completion. The body of the email notification is read from a text file in a specified location.

Essbase_Perl.pm is not limited to MaxL commands. It can also execute .bat and .cmd files.

As you can see, this program is highly configurable. More details to come tomorrow.