Kilo Mike

Saturday, August 29, 2015

Save Postgres execution plan in database table

Postgres database doesn't provide a built-in decent way to follow statement execution plans at different points in time as it is in Oracle, for example. Imagine an ETL query takes two hours during the night while during the day it's only two minutes. Is it a different execution plan generated? In Postgres it's pretty hard to find that out unless you're sitting there during the night and following up on it.

One actually pretty simple way to get it done could be with your ETL tool and store the execution plan output together with the generation time in a database table. Here I'm using Kettle, but the same would most likely be possible with every other software.

First create a table in the target database:

CREATE TABLE save_xplan_output
(
query_plan text,
x_plan_date timestamp without time zone
)

Next, create a "Table Input" step and write the following query:

Then add a "Get System Info" step to get the date and time:

Next, add the target table for the plan:

After creating the hops between elements your transformation would look like this:

After executing the transformation you can check the results:

You could put this kind of transformation somewhere in the ETL flow next to the problematic statement and you could be almost sure that the statement would have the same statement. If the execution plan differs then you can try doing something about the table statistics, the query itself. If not, then see what's going on with the environment, parallel activities etc.
Thanks for your attention!

Friday, August 14, 2015

OBIEE Page Caption Changes

Among many different OBIEE customization things one might also want to change the page captions that show up at the browser. Not only it would personalize the system, but it might also be helpful for a user who has his 10+ tabs open to find his OBIEE environment.
That's what I'm gonna show in this post and in this example I will just change the "Oracle BI(...)" default messages to "Test Environment".

Fortunately, you don't need to change any system javascripts and/or .css files and you can simply do this with adding or modifying your custom skin XML files.

First, you should either add or modify the already existing productmessages.xml file in your messages folder and have these kind of entries there:

<WebMessage name="kmsgProductPortal"><TEXT> Test Environment </TEXT></WebMessage>

<WebMessage name="kmsgProductAnswers"><TEXT>Test Environment - Answers</TEXT></WebMessage>

<WebMessage name="kmsgProductDelivers"><TEXT>Test Environment - Agents</TEXT></WebMessage>

<WebMessage name="kmsgProductAdvancedReporting"><TEXT>Test Environment - Publisher</TEXT></WebMessage>

<WebMessage name="kmsgProductGeneral"><TEXT>Test Environment</TEXT><HTML>Test Environment</HTML></WebMessage>

<WebMessage name="kmsgProductAnswers"><TEXT>Test Environment - Answers</TEXT></WebMessage>

Another file to modify or add is saw.catalog.xml file under the uicmsgs folder. You should have the following entries there:

<WebMessage name="kmsgCatalogWindowTitle"><TEXT>%0% - Test Environment - Catalog</TEXT></WebMessage><WebMessage name="kmsgCatalogStartingWindowTitle"><TEXT>Test Environment - Catalog</TEXT></WebMessage>

After the changes are done, you would need to reload the XML files from the Administration window in your OBIEE environment and the changes should be there.

Tuesday, August 11, 2015

Picture Addition for OBIEE Login Page

Here I'm gonna show how to add a nice background picture for the OBIEE login page.
After finding the desired picture you should place it in the following location:
"(...)\tmp\_WL_user\analytics_11.1.1\7dezjl\war\res\sk_blafp\login"

Next, you should take a backup of the login.css file located in the same directory and then open it up for modification. It's the first element "body" that you need to change. Take the following piece of code:

body
{
background-color: #fff;
background-image: url(background_blue_whitegradient_.png);
background-position: top left;
background-repeat: repeat-x;
margin-bottom: 0;
margin-left: 0;
margin-right: 0;
margin-top: 0;
}

And change it to(considering that the name of the new desired background is "background.jpg"):

body {
background-image: url(background.jpg);
background-position: top center;
background-color: #fff;
background-repeat: no-repeat;
background-size: 100%;
margin-left: 0;
margin-top: 0;
margin-right: 0;
margin-bottom: 0;
}

Restart the OBIEE servers and enjoy the result.

Of course, you might also want to customize OBIEE and change the Oracle logo or the brand name at the top left corner, but there are already plenty of sources describing that and I won't cover that here.

Thanks for watching!

Sunday, July 19, 2015

Event table bug

While implementing OBIEE physical cache clearance run into a weird behavior of the event table and the non-relevant row handling. At first couldn't really find out the reason and the cure, but in the end came to the root of the problem. I'll describe it in this entry.

I assume that you're familiar with the OBIEE event table that is used for clearing the cache entries of the database tables, so I'm not gonna go through that.

At first, let's insert two new rows in the table with the SYSDATE as the update time.

Now let's wait for the interval to pass and the entries to be deleted from cache. After a few minutes, when we select from the event table, we can see that instead of two entries there are four ones with "gs" being the value in OTHER_RESERVED column. Why?

Didn't find anything useful in Google, there were people who have had the same problem, but no solution to the problem was in the topic.
My attention was drawn by a sentence in Oracle documentation about the UPDATE_TS column. It said: "This needs to be a key (unique) value that increases for each row added to the event table."
So, I tried inserting the same rows, this time with an incremental SYSDATE value. Like this:

After the regular cache clearance, you can see that there are no entries in the table anymore.

After doing some additional testing, it did turn out that the event table really does handle rows with identical UPDATE_TS columns in an awkward way. From the nqquery.log it didn't seem that the cache entries for the corresponding tables with duplicate rows were continuously purged though.

So, it might not be advisable to use a simple SYSDATE as the update timestamp for the event table. Using CURRENT_TIMESTAMP in a single transaction also wouldn't help, thus you could create a custom procedure that receives the table name and inserts it into the table with the highest current UPDATE_TS value incremented with one second and use SYSDATE, if the table is empty. For example:

create or replace procedure add_to_event_table (database_name in varchar2, catalog_name in varchar2, schema_name in varchar2, table_name in varchar2)
as
max_date date;
begin
select NVL(max(update_ts+1/86400),SYSDATE) into max_date from S_NQ_EPT;
INSERT INTO S_NQ_EPT (UPDATE_TS, DATABASE_NAME, CATALOG_NAME, SCHEMA_NAME, TABLE_NAME)
VALUES (MAX_DATE, DATABASE_NAME, CATALOG_NAME, SCHEMA_NAME, TABLE_NAME);
COMMIT;
end;

Friday, July 10, 2015

Concatenate distinct row values of multiple columns effectively

Converting columns to rows, rows to columns, can sometimes be a bit of a headache for database developers. Recently run into a case where I needed to modify a dataset in order to have all the distinct row values from two separate columns for one ID concatenated in one row in the same separate columns. Here's a visual example of a small test dataset:

What I needed to get was:
ID                            A                             B
1                        Juliet/Krisjanis          Kilo/Mike
2                        Tango                      Mike
3                        Juliet                        Bravo/Charley

From the first look might seem easy to do, but actually isn't at all. You might to select the values from each column in a separate query and then join them by ID or have several nested queries or...

Here's a very nice and compact way of doing it without a need of several subqueries or similar struggles. Just use regepxp_replace and listagg functions together with a windowing function like this:

regexp_replace( listagg(trim(your_column), '/') within GROUP (
    ORDER BY your_column) OVER (PARTITION BY your_id) ,'([^/]+)(/\1)+', '\1')

So, here's how it looks on the previous dataset:

Thanks for your attention!

Wednesday, July 8, 2015

Customize OBIEE Export Options

The default export options at OBIEE 11g has a load of options and some of them might not even be really used. Therefore removing some of the unnecessary ones and restructuring the dialog could ease the life for users.

In this example I'm gonna show how to remove the old 2003 PPT and Excel, Web Archive and XML formats, rename Excel 2007 (to "Excel (formatted)"), PPT 2007 (to "PowerPoint") and Tab Delimited Format (to "Excel (raw data)") options and move them all in one column.

There are three separate places to export the report (dashboard, answers and the catalog) and each of these places require some separate changes.

First, let's do the changes for the export option names. If you have a customized skin, you would need to three files or modify the existing ones in the XML message folder. If not, you need to modify s_blafp skin XML files (viewmessages.xml, viewscontrolmessages.xml and uicmsgs/saw.catalog.xml).

Contents of viewmessages.xml:

<?xml version="1.0" encoding="utf-8"?>
<WebMessageTables xmlns:sawm="com.siebel.analytics.web/message/v1">
<WebMessageTable system="CompoundViewUI" table="LinkText">
<WebMessage name="kmsgEVCLinkDownloadDataTab">
<HTML>Excel (raw data)</HTML>
</WebMessage>
<WebMessage name="kmsgCatalogActionExportExcel">
<HTML>Excel (formatted)</HTML>
</WebMessage>
<WebMessage name="kmsgEVCLinkDownloadPowerpoint">
<HTML>PowerPoint</HTML>
</WebMessage>
</WebMessageTable>
</WebMessageTables>

Contents of viewscontrolmessages.xml:

<?xml version="1.0" encoding="utf-8"?>
<WebMessageTables sawm:systemMessages="false" xmlns:sawm="com.siebel.analytics.web/message/v1">
<WebMessageTable>
<WebMessage name="kmsgEVCDownloadLinks">
<HTML>
<script src="fmap:downloadguard.js" type="text/javascript"/>
<sawm:choose>
<sawm:when name="noMenu">
<a href="javascript:void(null);" name="SectionElements" onclick="NQWClearActiveMenu();Download('@{command}&path=@{path}&Format=excel&Extension=.xlsx'); return false">
<sawm:messageRef name="kmsgEVCLinkDownloadExcel"/>
</a>&nbsp;
<a href="javascript:void(null);" name="SectionElements" onclick="NQWClearActiveMenu();Download('@{command}&path=@{path}&Format=txt&Extension=.csv'); return false">
<sawm:messageRef name="kmsgEVCLinkDownloadData"/></a>&nbsp;
<a href="javascript:void(null);" name="SectionElements" onclick="NQWClearActiveMenu();Download('@{command}&path=@{path}&Format=mht'); return false">
<sawm:messageRef name="kmsgEVCLinkDownloadMHTML"/></a>
</sawm:when>
<sawm:otherwise>
<a href="javascript:void(null)" name="ReportLinkMenu" onclick="return NQWPopupMenu(event,'idDownloadLinksMenu@{jsStatePath}')" title="@{exporttitle}">
<sawm:messageRef name="kmsgEVCLinkDownload"/>
</a>
<div class="NQWMenu" id="idDownloadLinksMenu@{viewID}" onmouseover="NQWMenuMouseOver(event)">
<sawm:messageRef name="kuiMenuShadowBegin"/>
<a class="NQWMenuItem NQWMenuItemWIconMixin" href="javascript:void(null);" name="SectionElements" onclick="NQWClearActiveMenu();Download('@{command}&path=@{path}&Format=pdf&Attach=1'); return false">
<table cellpadding="0" cellspacing="0" class="MenuItemTable">
<tr>
<td class="MenuItemIconCell">
<img alt="" src="fmap:catalog/exporttopdf_ena.png"/>
</td>
<td class="MenuItemTextCell">
<sawm:messageRef name="kmsgEVCLinkDownloadPDF"/>
</td>
</tr>
</table>
</a>
<a class="NQWMenuItem NQWMenuItemWIconMixin" href="javascript:void(null);" name="SectionElements" onclick="NQWClearActiveMenu();Download('@{command}&path=@{path}&Format=excel2007&Extension=.xlsx');">
<table cellpadding="0" cellspacing="0" class="MenuItemTable">
<tr>
<td class="MenuItemIconCell">
<img alt="" src="fmap:catalog/exporttoexcel_ena.png"/>
</td>
<td class="MenuItemTextCell">
<sawm:messageRef name="kmsgCatalogActionExportExcel"/>
</td>
</tr>
</table>
</a>
<a class="NQWMenuItem NQWMenuItemWIconMixin" href="javascript:void(null);" name="SectionElements" onclick="NQWClearActiveMenu();return Download('@{command}&path=@{path}&Format=txt&Extension=.csv');">
<table cellpadding="0" cellspacing="0" class="MenuItemTable">
<tr>
<td class="MenuItemIconCell">
<img alt="" src="fmap:catalog/exporttoexcel_ena.png"/>
</td>
<td class="MenuItemTextCell">
<sawm:messageRef name="kmsgEVCLinkDownloadDataTab"/>
</td>
</tr>
</table>
</a>
<a class="NQWMenuItem NQWMenuItemWIconMixin" href="javascript:void(null);" name="SectionElements" onclick="NQWClearActiveMenu();return Download('@{command}&path=@{path}&Format=powerpoint2007&Extension=.pptx');">
<table cellpadding="0" cellspacing="0" class="MenuItemTable">
<tr>
<td class="MenuItemIconCell">
<img alt="" src="fmap:catalog/exporttoppt_ena.png"/>
</td>
<td class="MenuItemTextCell">
<sawm:messageRef name="kmsgEVCLinkDownloadPowerpoint"/>
</td>
</tr>
</table>
</a>
<a class="NQWMenuItem NQWMenuItemWIconMixin" href="javascript:void(null);" name="SectionElements" onclick="NQWClearActiveMenu();return Download('@{command}&path=@{path}&Format=csv&Extension=.csv');">
<table cellpadding="0" cellspacing="0" class="MenuItemTable">
<tr>
<td class="MenuItemIconCell">
<img alt="" src="fmap:catalog/exporttocsv_ena.png"/>
</td>
<td class="MenuItemTextCell">
<sawm:messageRef name="kmsgEVCLinkDownloadDataCSV"/>
</td>
</tr>
</table>
</a>
<sawm:messageRef name="kuiMenuShadowEnd"/>
</div>
</sawm:otherwise>
</sawm:choose>
</HTML>
</WebMessage>
</WebMessageTable>
</WebMessageTables>

Contents of saw.catalog.xml:

<?xml version="1.0" encoding="utf-8"?>
<WebMessageTables xmlns:sawm="com.siebel.analytics.web/message/v1"><WebMessageTable lang="en-us" table="Messages" system="saw.catalog">
<WebMessage name="kmsgCatalogActionExportTabDelimited"><TEXT>Excel (raw data)</TEXT></WebMessage>
</WebMessageTable></WebMessageTables>

This would do for the exports at dashboards and answers, but the catalog is a different story and in order to get that fixed you need to change two .js files.
Find the actionhandler.js and catalogaction.js files in (...)\web\appv2\res\b_mozilla\catalog and ${fmw_home}\user_projects\domains\airbit_domain\servers\bi_server1\tmp\_WL_user\analytics_11.1.1\7dezjl\war\res\b_mozilla\catalog directories and take a backup.
First, let's modify actionhandlerimpl.js. Find the following piece of code:

d = d.concat([saw.catalogaction.ActionId.EXPORT, saw.catalogaction.ActionId.EXPORT_PDF, saw.catalogaction.ActionId.EXPORT_EXCEL_GROUP, saw.catalogaction.ActionId.EXPORT_EXCEL2K, saw.catalogaction.ActionId.EXPORT_EXCEL, saw.catalogaction.ActionId.EXPORT_POWERPOINT_GROUP, saw.catalogaction.ActionId.EXPORT_POWERPOINT, saw.catalogaction.ActionId.EXPORT_POWERPOINT_2007, saw.catalogaction.ActionId.EXPORT_MHTML, saw.catalogaction.ActionId.EXPORT_DATAGROUP, saw.catalogaction.ActionId.EXPORT_CSV, saw.catalogaction.ActionId.EXPORT_TABDELIMITED, saw.catalogaction.ActionId.EXPORT_XML])

And replace it with this:

d = d.concat([saw.catalogaction.ActionId.EXPORT, saw.catalogaction.ActionId.EXPORT_PDF, saw.catalogaction.ActionId.EXPORT_EXCEL, saw.catalogaction.ActionId.EXPORT_TABDELIMITED, saw.catalogaction.ActionId.EXPORT_POWERPOINT_2007, saw.catalogaction.ActionId.EXPORT_CSV])

Next, let's modify catalogaction.js. Find these separate entries and remove:

EXPORT_EXCEL_GROUP: "exportExcelGroup",
EXPORT_EXCEL2K: "exportExcel2K",
EXPORT_POWERPOINT_GROUP: "exportPowerpointGroup",
EXPORT_POWERPOINT: "exportPowerpoint",
EXPORT_DATA: "exportData",
EXPORT_MHTML: "exportMhtml",
EXPORT_DATAGROUP: "exportDataGroup",
EXPORT_XML: "exportXML",

And remove this part of the code:

if (a) {this.addAction(b.EXPORT_EXCEL2K, "kmsgCatalogActionExportExcel2003", "catalog/exporttoexcel_ena.png", null, b.EXPORT_EXCEL_GROUP).addAction(b.EXPORT_EXCEL, "kmsgCatalogActionExportExcel2007", "catalog/exporttoexcel_ena.png", null, b.EXPORT_EXCEL_GROUP)
} else {
this.addAction(b.EXPORT_EXCEL2K, "kmsgCatalogActionExportExcel", "catalog/exporttoexcel_ena.png", null, b.EXPORT)
}

And then find:

addAction(b.EXPORT_POWERPOINT, "kmsgCatalogActionExportPowerpoint2003", "catalog/exporttoppt_ena.png", null, b.EXPORT_POWERPOINT_GROUP).addAction(b.EXPORT_POWERPOINT_2007, "kmsgCatalogActionExportPowerpoint2007", "catalog/exporttoppt_ena.png", null, b.EXPORT_POWERPOINT_GROUP).addAction(b.EXPORT_DATA, "kmsgCatalogActionExportData", "catalog/exporttocsv_ena.png", null, b.EXPORT).addAction(b.EXPORT_MHTML, "kmsgCatalogActionExportMhtml", "catalog/exporttomhtml_ena.png", null, b.EXPORT).addAction(b.EXPORT_PDF, "kmsgCatalogActionExportPdf", "catalog/exporttopdf_ena.png", null, b.EXPORT).addAction(b.EXPORT_CSV, "kmsgCatalogActionExportDefaultCSV", "catalog/exporttocsv_ena.png", null, b.EXPORT_DATAGROUP).addAction(b.EXPORT_TABDELIMITED, "kmsgCatalogActionExportTabDelimited", "catalog/exporttocsv_ena.png", null, b.EXPORT_DATAGROUP).addAction(b.EXPORT_XML, "kmsgCatalogActionExportXML", "catalog/exporttoxml_ena.png", null, b.EXPORT_DATAGROUP)

And change it to this:

addAction(b.EXPORT_EXCEL, "kmsgCatalogActionExportExcel", "catalog/exporttoexcel_ena.png", null, b.EXPORT).addAction(b.EXPORT_POWERPOINT_2007, "kmsgCatalogActionExportPowerpoint", "catalog/exporttoppt_ena.png", null, b.EXPORT).addAction(b.EXPORT_PDF, "kmsgCatalogActionExportPdf", "catalog/exporttopdf_ena.png", null, b.EXPORT).addAction(b.EXPORT_CSV, "kmsgCatalogActionExportDefaultCSV", "catalog/exporttocsv_ena.png", null, b.EXPORT).addAction(b.EXPORT_TABDELIMITED, "kmsgCatalogActionExportTabDelimited", "catalog/exporttoexcel_ena.png", null, b.EXPORT)

This should be it. Now restart the presentation services server and see the results.

Thanks for watching! Hopefully, you will find this useful and helpful.