Category Archives: Database

Are Daily Backups Really Sufficent?

Monday afternoon we had a critical failure of an Oracle database at work. Within a few minutes of the fault taking place, I started seeing block corruption errors whilst I was reviewing some information in the production environment. At this stage, I was thinking that we might have dropped a disk in the SAN but referred it onto our database administrator to rectify it.

As is quite common, our environment consists of multiple Oracle 10g RAC nodes connected into a shared data source. The shared data source in this instance is a SAN, where we have a whole bunch of disks configured in groups for redundancy and performance. As soon as the database administrator became involved, it became apparent that we didn’t drop a single disk but had in fact lost access to an entire group of disks within the SAN.

Due to the manner in which the SAN and Oracle are configured, we were not in a position where running in a RAID environment was going to help. If we had dropped a single disk or a subset of disks from any group within the SAN, everything would have been fine; unfortunately we dropped an entire disk group. The end result of this was that we were forced to roll back our database to the previous nights backup.

The following days have been spent recovering the lost days data through various checks and balances; but it takes a lot of time and energy from everyone involved to make this happen. We’ve been fortunate enough to trade for several years without ever needing to roll back our production database due to some sort of significant event; which I suppose we should be thankful for.

After three years without performing a production disaster recovery, had we become complacent about data restoration and recovery as haven’t really needed it before? I believe that since we haven’t had a requirement to perform a disaster recovery for some three years, that our previous data recovery guidelines have now become out of date. Whilst a daily backup may have been more than sufficient for this particular database two or three years ago, the business has undergone significant growth since that time. The daily changeset for this database is now significant enough that, whilst having a daily backup is critical – it requires significant amounts of work to recover all of the data in a moderate time frame.

As a direct result of this disaster, we’re going to be reviewing our data recovery policies shortly. The outcome of that discussion will most likely be that we require higher levels of redundancy in our environment to reduce the impact of a failure. Whilst it would be ideal to have an entire copy of our production hardware, it probably isn’t going to be a cost effective solution. I’m open to suggestions about what sort of data recovery we implement, however I think that having some sort of independent warm spare may win out.

What have we learned out of this whole event:

  • daily backup of data is mandatory
  • daily backup of data may not be sufficient
  • verify that your backup sets are valid, invalid backup data isn’t worth the media it is stored on
  • be vigilant about keeping data recovery strategies in step with business growth and expectations

Maybe periodic disasters are actually healthy for a business? Whilst every business strives to avoid any sort of down time, I expect that as a direct result of the typically high availability of certain systems that disaster recovery isn’t put through its paces often or rigorously enough; which may result in longer downtimes or complete loss of data when an actual disaster recovery is required.

Tech Ed 2007, Day 3 Wrap Up

Yesterday I looked into the building of Background Motion using the Composite Web Block, the Enterprise Library and putting all of the different .NET 3.x technologies together in a demonstration product named Dinner Now. Today was focused around SQL Server 2005 performance, optimisation and scalability followed by .NET language pragmatics.

Writing Applications That Make SQL Server 2005 Fly

Shu Scott presented about writing applications that make SQL Server 2005 fly, however I don’t think that name reflected the presentation all that well. The talk would have been better titled ‘Understanding The Cost Based Optimiser To Make SQL Server 2005 Fly’. None the less, Shu raised a lot of great points in her presentation and some of them I thought interesting are below:

  • Don’t use a function passing in a column as a parameter within a query, such as in a WHERE clause. SQL Server 2005 calculates statistics for a table per column, so as soon as you use a function on the column the statistics are unusable. The off shoot of this is that SQL Server 2005 can massively under or over estimate the selectivity of a table which on a complex-ish based query can dramatically change the query plan that SQL Server will choose.
  • Don’t alter an input parameter to a function or stored procedure after the procedure has started. Shu didn’t specify exactly why this is the case, however after investigating it further on the internet; it is related to the point below regarding parameter sniffing.
  • Avoid using local variables within a function or procedure in WHERE statements. During the presentation, I didn’t get a clear understanding of why this was sub-optimal, however after some research online it is caused by the cost based optimiser having to use a reduced quality estimate for the selectivity of the table. You can avoid this problem by supplying the OPTION(RECOMPILE> hint, use a literal instead of a local variable if possible, parameterise the query an accept the value via an input parameter.
  • Use automatic statistics, if you have a requirement to not use it – disable it on a per table basis if possible as having quality statistics for your database is vital in the cost based optimiser doing its job.
  • Do parameterise your queries where they are common, with a lot of reuse and are hit often. Do not parameterise queries that are ad-hoc or long running. Presumably there is no gain in parameterising a long running query as the server is already going to be spending significant time processing the query, in which case the few milliseconds the server spends generating a query plan won’t be noticed.
  • Be aware of parameter sniffing, which is where SQL Server uses the values of the input values to a function/procedure to produce the query plan. This is normally a very good thing, however if the cached plan happens to represent an atypical set of input values – then it is likely that the performance of a typical query is going to be severely impacted.
  • Look to utilise the INCLUDE keyword when creating non-clustered indexes. The INCLUDE keyword allows you to extend past the 900 byte limit on the index key and also allows you to include previously disallowed column types within the index (ex: nvarchar(max)). This type of index is excellent for index coverage, as all columns identified are stored within the index in leaf nodes, however only the key columns enforce the index type.
  • If you are unable to edit an SQL statement for some reason, consider using the plan guides. A plan guide is essentially a set of option hints for the query, however you aren’t editing the query itself to apply them. You configure the plan guides for a stored procedure, function or an individual statement and when it is matched SQL Server 2005 will automatically apply the suggested guides to the statement.
  • In a similar fashion to the plan guide, there is a more complex option called USE PLAN which lets you supply an actual execution plan to the SQL statement, again without editing the SQL statement directly. Essentially, you extract the XML representation for the execution plan you would prefer to have execute and supply that to the SQL statement. If you have skewed data sets, this would be a good option to guarantee consistent access speed for a particular query. Using the skewed data sets as an example, it would be possible to have SQL Server cache a query plan which represents the atypical data and as such performs very poorly for the majority of the typical data. Supply the query plan to the SQL statement can avoid that happening. It is worth noting though, if you do supply a query plan you would need to revisit the stored plan periodically to make sure that it still reflects the most efficient access path for your particular data set.

Implementing Scale Out Solutions With SQL Server 2005

This presentation was about scaling SQL Server 2005 out, such that you’re able to continue adding more and more servers to the mix to distribute the load across. I had previously read the majority of the information covered, however I learned about two features named the SQL Server Service Broker and Query Notifications.

  • Linked servers let you move whole sections of a database onto another server and you tell the primary server where the other data resides. Linked servers are transactionally safe, however will perform only as fast as the slowest component within the linked server group.
  • Distributed Partitioned Views allows you to move subsets of a tables data across servers and uses linked servers behind the scenes to communicate between servers. A partition might be as simple as customers 1 through 100,000 in partition A and 100,001 through 200,000 in partition B and so on.
  • SQL Server Shared Database (SSD) allows you to attach many servers to a single read only copy of the database, which might be a great way of increasing performance for a heavily utilised reporting server with relatively static data. Unfortunately, the servers reading from the database need to be detached to refresh the database but this could be managed in an off peak period to reduce impact.
  • Snapshop Replication snapshots an entire database and replicates it into the subscribers. Snapshot replication isn’t used a lot as it’s data and resource intensive. It is most commonly used to set up a base system and then enable merge replication to bring it up to date with the publisher or to refresh an infrequently changing database periodically.
  • Merge Replication tracks the changes to the data on the publishers and bundles them together, only sending the net changes when appropriate. Merge replication supports bi-directional replication and it also implements conflict resolution as well, however there is no guarantee of consistency as the changes aren’t necessarily being replicated in a near real time environment.
  • Transaction Replication sends all changes to all subscribers and is transactionally safe. If there were a lot of DML taking place in a database, there would be considerable overhead for using transactional replication as a simple UPDATE statement which might effect 100 rows locally is sent to the subscribers as 100 independent SQL statements; in case some or all of the subscribes have additional data that the publisher does not.
  • Peer To Peer (P2P) Replication is a variation of transactional replication however it requires that each peer be the master of it’s own data so as to avoid key read/write problems across servers and consistency issues. As an example, all of the Brisbane office writes its changes into server A, while Sydney writes its changes into server B. By making sure that each server ‘owns’ its respective block of data, it is then possible and safe to replicate data between all peers safely.
  • SQL Server Service Broker (SSB) provides a reliable asynchronous messaging system to SQL Server 2005, that allows you to pass messages between application either within the same database, same server or distributed over many servers and databases. The service broker doesn’t do the work for you, however it does provide the plumbing to make developing your system a whole lot simpler. Using the service broker, it would even be possible to send messages from one service into another service on a different machine; might be useful to help keep different pieces of information up to date in a vastly distributed database set up when replication doesn’t quite suit the purpose.
  • Query Notification, as it suggests is a notification system which is used to notify clients or caches that they need to update certain data. Once again, the query notification doesn’t do the updating – it merely provides the event to tell you do perform your own action. The Query Notification engine utilises the service broker under the hood.
  • Data Dependent Routing isn’t a SQL Server feature but more of an architectural design pattern. Using Data Dependent Routing, the client (whatever it is), knows a little bit about the storage system and optimistically seeks out the data store which is likely to return the best performance.

.NET Programming Language Pragmatics

Joel Pobar presentated on .NET programming language pragmatics and contrasted some of the recent developments in that space. At the start of the talk, he pointed out that there are generally three types of programming languages – static, dynamic and functional. The original version of the .NET Common Language Runtime was based around a static environment and has recently been enhanced to support functional programming and more recently dynamic.

The dynamic programming languages are handled through a new component, the Dynamic Language Runtime which sits on top of the existing CLR. The new Dynamic Language Runtime has allowed people to build IronPython and IronRuby, which are implementations of those particular languages sitting over the top of the .NET CLR.

Outside of the fact that it means you’ll be able to run a Python script in the native Python runtime or inside of the .NET DLR, which is just plain cool; the biggest picture here is that the .NET CLR is being enhanced and soon we’ll have a new super language (LISP advocates, back in your boxes!) which will support all of the current types of programming languages at once.

The presentation was fantastic and it is exciting to hear Joel present as he is so passionate about the field. In fact, I would go as far to say that his enthusiasm for his work is infectious; it is hard to walk away from one of his presentations and not have at least some of his excitement and enthusiasm rubbed off on you.

I’ve heard on the grape vine that Joel might be able to present at the one of the next Gold Coast .NET User Groups, can’t wait if he does!


It’s a sign you need either:

  1. more sleep; or
  2. more coffee

when you find code committed into a source code repository that you don’t remember writing and realise when you find it that it was really handy that you wrote that code at some obscure hour of the morning a week ago.

Learning How To Scale An ASP.NET Application

For the last nine months, the development team at Stella Hospitality Group have been working on integrating a new piece of software into the enterprise. Throughout that process, we’ve come up against various stumbling blocks and subsequently learned how to climb over them.

One of the interesting parts of this project involved learning how to scale an ASP.NET web application. Unlike most other pieces of development we’ve previously worked on, we didn’t have access to hardware and services that were capable of delivering smoking performance (read: Oracle 10g clustered using RAC). As a by product of the constraints which were placed on us, scaling the new web application proved a little harder than it first looked.

Over the course of the next few weeks, I’ll be posting about various steps which we’ve taken to scale our ASP.NET application. Some of the points are hinged in the physical world, others operational and of course technical as well. Items which come to mind immediately include:

  • load balancers
  • clustering physical servers
  • clustering web servers
  • web gardens
  • user interface process control
  • session handling
  • web services & XML
  • spike testing

Oracle Dynamic SQL Using The DECODE Function

When developing most applications, you end up having to interact with a database. If you’re application is large enough and warrants the design and effort, it will no doubt be tiered as well. If your application is tiered, one of your layers will probably include a database API. A database API is used to restrict access to the database, a funnel which all requests from your application must pass through. Implementing this additional layer of abstraction normally yields in higher performance, increased security and easier maintenance.

In most cases, you’re application code will require a fairly standard interface into your database to retrieve information. A simple way to achieve that consistent interface is using the DECODE function in your PL/SQL procedures and functions. There are always cases where you need to do something special, and in those cases it is probably a perfect scenario for this methods counter-part – the string building technique.

For a simple but practical example, consider the following table definition:

  1. CREATE TABLE MyTable (
  2. PrimaryKeyID NUMBER NOT NULL,
  3. ForeignKeyID NUMBER NOT NULL,
  4. Name VARCHAR2 (255) NOT NULL
  5. )

As I have mentioned before, the Oracle DECODE function provides similar functionality to an IF-THEN-ELSE code block. Lets examine an example which relates to MyTable above.

  2. (pDetails OUT grcDetails,
  3. pPrimaryKeyID IN MyTable.PrimaryKeyID%TYPE DEFAULT NULL,
  4. pForeignKeyID IN MyTable.ForeignKeyID%TYPE DEFAULT NULL,
  5. pName IN MyTable.Name%TYPE DEFAULT NULL)
  6. IS
  7. BEGIN
  8. OPEN pDetails FOR
  9. SELECT *
  10. FROM MyTable
  11. WHERE PrimaryKeyID = DECODE(pPrimaryKeyID, NULL, PrimaryKeyID, 0, PrimaryKeyID, pPrimaryKeyID)
  12. AND ForeignKeyID = DECODE(pForeignKeyID, NULL, ForeignKeyID, 0, ForeignKeyID, pForeignKeyID)
  13. AND LOWER(Name) LIKE DECODE(pName, NULL, LOWER(Name), '%'||LOWER(pName)||'%');
  14. END;

Working top down, the second line declares pDetails (the parameter name) as being an OUT parameter. As the name suggests, an OUT parameter is a way for you to pass information out of your procedure. In some aspects it would be similar to returning a value out of a typical programming function. The difference to an OUT parameter is that you can have many OUT parameters in a single procedure, where as a typical programming function can only return a single value at a time.

The next thing you’ve probably noticed is this weird thing in there, grcDetails. Just as the IN parameters below it, grcDetails is the type of the parameter. It would be akin to defining a paramter in a traditional function as accepting an integer or string type. In this case, grcDetails is declared in the package header:


The important thing to notice about grcDetails is that it is a REF CURSOR and its type is the ROWTYPE of MyTable. The benefit of defining it as a ROWTYPE is that if tomorrow the definition of the table changed in some way, you don’t have to worry about changing any code to match the new table definition – it simply continues to work. This is the same reason why the INPUT parameters above have their type defined against the field that they represent. Of course, if you are passing in or out a value which isn’t based on a field – you would simply define it as a standard PL/SQL type.

Next, you have probably noticed that each IN parameter has a DEFAULT value. As with most programming languages, providing a DEFAULT value allows the programmer to optionally supply that parameter. For our example, this is a pivotal point as it simplifies the use of this method; more on that later.

Moving on to the actual SQL statement, the flexibility of the DECODE function starts to show. Since the DECODE function acts like an IF-THEN-ELSE block, you’re actually seeing an inline IF-THEN-ELSE IF-ELSE block per field. Lets take a practical example of that statement and assume we passed in pPrimaryKeyID with a value of 1, while the other two input parameter’s are not passed in (thus taking their default value of NULL). The SQL statement would be parsed and executed as follows:

  1. SELECT *
  2. FROM MyTable
  3. WHERE PrimaryKeyID = 1
  4. AND ForeignKeyID = ForeignKeyID
  5. AND LOWER(Name) LIKE LOWER(Name);

Of course, the net effect of that SQL statement is that the first WHERE condition is used, while the second and third conditions are nullified as the left and right half of each expression are equivalent. So by passing in a NULL value for the pForeignKeyID and pName parameters, they are effectively removed from the SQL statement by evaluating to themselves. Of course, you could also pass in two, three or none of the parameters to the procedure as well. In which case you would get either a restricted set from the statement or all records in the table.

As mentioned above, the DEFAULT value of each IN parameter is critical to this methods success. By providing a default value, the programmer no longer needs to call the procedure with all parameters; all, some or none are also perfectly valid combinations. Since the input parameter’s default value is NULL, the following DECODE function call is all that is required to negate the parameter in the WHERE clause:

  1. PrimaryKeyID = DECODE(pPrimaryKeyID, NULL, PrimaryKeyID, pPrimaryKeyID)

By now, you have probably noticed some extra parameters in the example GET procedure listed above. The additional values are used to exclude any other ‘not important’ values from the statement. In the example procedure, NULL and the value 0 are considered unimportant; whilst all other values are considered useful.

If you’re wondering why you’d want to exclude other values, it might be to make another section of your application simpler. Some programming languages don’t support nullable primitive types. If you use such a language and you intend to pass in all parameters to your procedure in all circumstances (this is the ‘simpler’ above); then all values will have a value of some sort. In the case of a primitive such as an integer, you might find that its default uninitialised value is zero. If that is the case and you don’t require the ability to filter on a zero value, then excluding it within the DECODE function makes things simpler.

Don’t think you’re limited to using this method on SELECT statements, it will work a treat on DELETE too. Consider the following DEL procedure:

  2. (pRowCount OUT NUMBER,
  5. IS
  6. BEGIN
  9. FROM RoomTypes
  10. WHERE RoomTypeID = DECODE(pRoomTypeID, NULL, RoomTypeID, 0, RoomTypeID, pRoomTypeID)
  11. AND BuildingID = DECODE(pBuildingID, NULL, BuildingID, 0, BuildingID, pBuildingID);
  12. ELSE
  13. RAISE_APPLICATION_ERROR(-20001, 'At least one parameter must be supplied');
  14. END IF;
  15. pRowCount := SQL%ROWCOUNT;
  16. END;

There is a caveat to using this method for building dynamic SQL within Oracle, it cannot handle columns which are nullable. If you consider the use of the DEFAULT value on all of the input parameters, it will become clear. Within Orcale PL/SQL, it is not possible to use an equality (=) operator to compare a NULL value. As a simple example, take the two simple SQL statements:


The first SQL statement above will return the expected row from the DUAL table, while the second statement will return no results as you cannot compare the NULL value in that manner. If a comparison against a NULL is required, it must be handled using the IS NULL clause.

Other than not being able to use this method against columns which are nullable, its a really convenient way to write dynamic SQL in Oracle. Next time you need a little flexibility and you don’t want to go down the string building path, try using the Oracle DECODE function to produce your dynamic SQL.