Skip to main content

Using LOG ERRORS as a testing mechanism

I recently heard a very interesting story about a use for LOG ERRORS: namely, to verify the correctness (and improve the performance) of a problematic SQL statement.

I thought you might enjoy hearing about it.

Greg Belliveau of Digital Motorworks buttonholed me at ODTUG's Kscope14 conference with this report:

As part of a data mart refresh, we had an insert statement that refreshed a FACT table, and over the years had gotten to the point where it took well over 3 hours to complete. 

There was a NOT IN clause in the statement, and we were pretty sure that was the cause of the degenerating performance. Why was it there? Our best guess was so that the statement would not fail in case someone ran the refresh for a date range that had already ran...even though there was code that checked and prevented that from happening. 

[Steven: In other words, programmer's insurance. "Just in case, let's do this." That's always problematic in software. Is that a real "case"? Will it ever happen? What are the consequences of including this insurance? Will we document it so that others coming along later can figure out why this weird code is here? Usually not.]

Not wanting to lose the "safety net" that was in place, we decided to try the LOG ERRORS feature you had mentioned in your session at Oracle Open World in 2010. We removed the NOT IN clause and added LOG ERRORS. The insert statement then ran (and continues to run to this day) in roughly one minute (down from three hours!).

Oh, and there's never been a single row inserted in the error table!

Nice, nice, very nice.

It's always a bit scary to mess with code that's been around for years and (since it's been around for years) is a gnarly bunch of logic. SQL, with its set orientation, can be even more off-putting than, say, a PL/SQL function.

So, of course, the solution is to build a comprehensive automated regression test script so that you can compare before and after results.

Yes, but almost no one will (ever) do that. So we do what we can. 

And in this case, Greg and his team came up with a very creative solution:

We are pretty sure NOT IN is causing the performance problem, but we also cannot afford to remove the clause and have the statement fail.

So we'll take it out, but add LOG ERRORS to ensure that all inserts are at least attempted.

Then we can check the error log table afterwards to see if the NOT IN really was excluding data that would have caused errors.

In Greg's case, the answer was "Nope". So far as they can tell, the NOT IN was unnecessary.

OK, but maybe it will be necessary a week or month or year from now. What then?

Well, the bad data that should have been excluded will still be excluded (the insert will fail), and then the job can check the log table and issue whatever kind of report (or alarm) is needed.

So LOG ERRORS as part regression test, part performance monster.

Nice work, Greg.

Nice work, Oracle.


Comments

  1. Hi Steven.
    I've recently discovered that Data Pump uses LOG ERRORS under the hood to implement the feature of "skip constraint errors" during import: http://www.db-oriented.com/2014/07/19/impdp-which-rows-failed
    It's nice that Oracle uses its own features internally.
    Thanks,
    Oren.

    ReplyDelete
  2. Hello Steven & Oren,

    Very nice, thanks to Oren for his comment :) :)

    What a small world ... to find Oren's post on a blog that we both follow ...
    and we also know each other from the last Israeli Oracle Week 2013 ...
    we only missed Steven to join us there, am I right, Oren ?

    Regarding the automatic error logging performed during IMPDP ...
    I read the post on your blog and saw the nice trick for temporarily preserving the
    error logging table ... but, I wonder, why doesn't Oracle always preserve this table
    as a default behavior ... in fact ... it is NOT of much use to fill a table with data
    and then just drop it before anyone has the chance to use that data ...

    I wonder whether other tricks are possible, like, for example,
    using a DDL event trigger to catch the creation of the ERRDP$ table,
    and then create another "shadow" table for it and a trigger to copy the data
    from ERRDP$ into the shadow table which will remain ...

    I cannot but express my hope that we all will meet one day :) :)

    Thanks a lot & Best Regards,
    Iudith

    ReplyDelete

Post a Comment

Popular posts from this blog

Get rid of mutating table trigger errors with the compound trigger

When something mutates, it is changing. Something that is changing is hard to analyze and to quantify. A mutating table error (ORA-04091) occurs when a row-level trigger tries to examine or change a table that is already undergoing change (via an INSERT, UPDATE, or DELETE statement). In particular, this error occurs when a row-level trigger attempts to read or write the table from which the trigger was fired. Fortunately, the same restriction does not apply in statement-level triggers.

In this post, I demonstrate the kind of scenario that will result in an ORA-04091 errors. I then show the "traditional" solution, using a collection defined in a package. Then I demonstrate how to use the compound trigger, added in Oracle Database 11g Release1,  to solve the problem much more simply.

All the code shown in this example may be found in this LiveSQL script.

How to Get a Mutating Table Error

I need to implement this rule on my employees table:
Your new salary cannot be more than 25x th…

Table Functions, Part 1: Introduction and Exploration

August 2018 update: please do feel encourage to read this and my other posts on table functions, but you will learn much more about table functions by taking my Get Started with PL/SQL Table Functions class at the Oracle Dev Gym. Videos, tutorials and quizzes - then print a certificate when you are done!


Table functions - functions that can be called in the FROM clause of a query from inside the TABLE operator - are fascinating and incredibly helpful constructs.

So I've decided to write a series of blog posts on them: how to build them, how to use them, issues you might run into.

Of course, I am not the first to do so. I encourage to check out the documentation, as well as excellent posts from Adrian Billington (search for "table functions") and Tim Hall. Adrian and Tim mostly focus on pipelined table functions, a specialized variant of table functions designed to improve performance and reduce PGA consumption. I will take a look at pipelined table functions in the latte…

Quick Guide to User-Defined Types in Oracle PL/SQL

A Twitter follower recently asked for more information on user-defined types in the PL/SQL language, and I figured the best way to answer is to offer up this blog post.

PL/SQL is a strongly-typed language. Before you can work with a variable or constant, it must be declared with a type (yes, PL/SQL also supports lots of implicit conversions from one type to another, but still, everything must be declared with a type).

PL/SQL offers a wide array of pre-defined data types, both in the language natively (such as VARCHAR2, PLS_INTEGER, BOOLEAN, etc.) and in a variety of supplied packages (e.g., the NUMBER_TABLE collection type in the DBMS_SQL package).

Data types in PL/SQL can be scalars, such as strings and numbers, or composite (consisting of one or more scalars), such as record types, collection types and object types.

You can't really declare your own "user-defined" scalars, though you can define subtypes from those scalars, which can be very helpful from the perspective…