RMAN does (not?) backup empty blocks | what is NULL COMPRESSION ? | what is BINARY COMPRTESSION ?| what is UNUSED BLOCK COMPRESSION ?

January 27, 2006

In my class this week I have pointed out an important point about RMAN which is commonly misunderstood by many DBAs:
Some believe that RMAN does not backup empty blocks at all.
Others believe that RMAN does not backup empty block above the HIGH WATER MARK.
TRUE IS:
RMAN in ORACLE 9i and before does not backup empty blocks which have never been used!!!
in other words: RMAN does not backup empty blocks above the high watermark which are not formated!!
This is called NULL COMPRESSION.

In Oracle 10g Release 2 RMAN does not backup unused blocks (empty blocks below the High watermark).
This new feature is called UNUSED BLOCK COMPRESSION, see online doku 10gr2:
RMAN now creates more compact backups of datafiles, by skipping datafile blocks that are not currently used to store data. In previous releases, RMAN only supported NULL compression, which skipped space in datafiles that had never been allocated. No extra action is required on the part of the DBA to use this feature.

AS of ORACLE 10g RMAN can even compress backupsets. (Binary backupset compression), which can reduce the space needed for the backup to one fifth of size of an uncompressed backup.

RMAN> configure device type disk backuptype to compressed backupset;# make it the default!orRMAN> backup as compressed backupset database;-- only compress this single one 


Here is my demo for NULL COMPRESSION from my class:
1. I create a new tablespace with one datafile of size 10m.
2. I backup the datafile with RMAN, => very small backupset!
3. I create a table in this datafile and insert 1000000 rows.
4. I take another backup with RMAN, => big backupset.
5. I delete the rows and commit;
6. I take another backup with RMAN, => still big backupset.
7. I truncate the table, high watermark is down to first block in first extent!!
8. I take another backup with RMAN, => still big backupset.

Here is the demo-code:

sys@orcl9iR2 SQL> CREATE TABLESPACE lutz_test
2 DATAFILE ‘/opt/oracle/oradata/orcl/lutz_test_01.dbf’ SIZE 10m
sys@orcl9iR2 SQL> SELECT file#
2 FROM v$datafile
3 WHERE name = ‘/opt/oracle/oradata/orcl/lutz_test_01.dbf’ ;

FILE#
———-
9
sys@orcl9iR2 SQL> SELECT bytes, blocks, user_bytes, user_blocks
2 FROM dba_data_files
3 where FILE_ID=9;
BYTES BLOCKS USER_BYTES USER_BLOCKS
———- ———- ———- ———–
10485760 1280 10420224 1272
sys@orcl9iR2 SQL> SELECT (10485760-10420224)/1024 FROM dual;
(10485760-10420224)/1024
————————
64
# 64K contain extent bitmap for tablespace
[oracle@KR11-KL1 oracle]$ rman target /
Recovery Manager: Release 9.2.0.7.0 – Production
Copyright (c) 1995, 2002, Oracle Corporation. All rights reserved.
connected to target database: ORCL (DBID=1108329042)
RMAN> BACKUP DATAFILE 9;
channel ORA_DISK_1: finished piece 1 at 27-JAN-06
piece handle=/home/oracle/BACKUP/RMAN/0nh9sopv_1_1.rman comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 27-JAN-06
[oracle@KR11-KL1 oracle]$ ll /home/oracle/BACKUP/RMAN/0nh9sopv_1_1.rman
-rw-r—– 1 oracle oinstall 98304 Jan 27 06:50 /home/oracle/BACKUP/RMAN/0nh9sopv_1_1.rman
sys@orcl9iR2 SQL> SELECT 98304/1024 FROM dual;
98304/1024
———-
96
# size of backupset with empty datafile

sys@orcl9iR2 SQL> ! ls -l /opt/oracle/oradata/orcl/lutz_test_01.dbf
-rw-r—– 1 oracle oinstall 10493952 Jan 27 06:50 /opt/oracle/oradata/orcl/lutz_test_01.dbf
sys@orcl9iR2 SQL> SELECT 10493952/1024 FROM dual;
10493952/1024
————-
10248
# size of empty datafile on disk

sys@orcl9iR2 SQL> CREATE TABLE lutz_tab
2 ( sp1 NUMBER) TABLESPACE lutz_test;
sys@orcl9iR2 SQL> BEGIN
2 FOR i IN 1..1000000 LOOP
3 INSERT INTO lutz_tab values(i);
4 END LOOP;
5 END;
6 /
PL/SQL procedure successfully completed.
sys@orcl9iR2 SQL> COMMIT;
Commit complete.

RMAN> BACKUP DATAFILE 9;
channel ORA_DISK_1: finished piece 1 at 27-JAN-06
piece handle=/home/oracle/BACKUP/RMAN/0ph9spfo_1_1.rman comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
Finished backup at 27-JAN-06
[oracle@KR11-KL1 oracle]$ ls -l /home/oracle/BACKUP/RMAN/0ph9spfo_1_1.rman
-rw-r—– 1 oracle oinstall 9535488 Jan 27 07:02 /home/oracle/BACKUP/RMAN/0ph9spfo_1_1.rman
sys@orcl9iR2 SQL> SELECT 9535488/1024 FROM dual;
9535488/1024
————
9312
# size of backupset with datafile after inserts

sys@orcl9iR2 SQL> DELETE FOM lutz_tab;
1000000 rows deleted.
sys@orcl9iR2 SQL> COMMIT;
Commit complete.
RMAN> BACKUP DATAFILE 9;
channel ORA_DISK_1: finished piece 1 at 27-JAN-06
piece handle=/home/oracle/BACKUP/RMAN/0rh9spll_1_1.rman comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 27-JAN-06
[oracle@KR11-KL1 oracle]$ ls -l /home/oracle/BACKUP/RMAN/0rh9spll_1_1.rman
-rw-r—– 1 oracle oinstall 9535488 Jan 27 07:05 /home/oracle/BACKUP/RMAN/0rh9spll_1_1.rman
# size of backupset with datafile AFTER DELETING all rows

sys@orcl9iR2 SQL> TRUNCATE TABLE lutz_tab;
Table truncated.
RMAN> BACKUP DATAFILE 9;
channel ORA_DISK_1: finished piece 1 at 27-JAN-06
piece handle=/home/oracle/BACKUP/RMAN/0th9spok_1_1.rman comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 27-JAN-06
[oracle@KR11-KL1 oracle]$ ls -l /home/oracle/BACKUP/RMAN/0th9spok_1_1.rman
-rw-r—– 1 oracle oinstall 9535488 Jan 27 07:07 /home/oracle/BACKUP/RMAN/0th9spok_1_1.rman
# size of backupset with datafile AFTER TRUNCATING
!!! empty formatted blocks backed up!!!

Related Metalink Notes:
Note: 247705.1 RMAN BACKUP COMPRESSION
Note: 255973.1 Oracle10g Recovery Manager – Backup Compression Methods


Flashback any error with Oracle 10g release 2?

January 23, 2006

(this is my article published in SOUG newsletter 1/2006)

Oracle 10g comes with a number of flashback technologies. You can use these functionalities to go back to a point in time in the past on different levels in the event of a logical error. You can, on row-, table- and even on database level undo human errors now.

A variety of functionalities are summarized under the name of flashback, which in fact use very different technologies.
This article deals with all of these functionalities.

Let us first take a short look at the history of flashback functionalities.

Starting with Oracle 9i already the first flashback functionalities were made available:
The Oracle 9i Release 1 package DBMS_FLASHBACK enables a DBA to reset a session to a point in time in the past:
exec dbms_flashback.enbale_at_time (sysdate-10/(24*60))
resets the current session to a point in time 10 minutes ago. From now on, when the session opens a cursor, it will see the read consistent image of the committed information in the database as of 10 minutes ago. This was not possible for a session which is connected as sys. With Oracle 10g it is possible.

Flashback query, an Oracle 9i Release 2 functionality, enables us to select values from a table as of a point in time in the past:
SELECT *
FROM hr.employees
AS OF TIMESTAMP (sysdate-10/(24*60));

And already in Oracle 9i it was possible to create a table as a copy of another table as of a point in time in the past.

CREATE TABLE hr.emps
AS
SELECT *
FROM hr.employees
AS OF TIMESTAMP
(sysdate-10/(24*60));

This functionality was improved in Oracle 10g with a new technology (further details at the bottom of this page under flashback table).
All these Oracle 9i flashback functionalities use data from undo segments. Thus, precondition for all of them to work is that all before-images needed for a query must be still available. The new in Oracle 9i dynamic initialization parameter UNDO_RETENTION provides us with the possibility, to tell the server how long before images should be preserved at least, even after commit. In Oracle 9i there is no guarantee that all these images will be available for the period of undo_retention. The server will try its best, however under space pressure in the undo tablespace it will overwrite before images even before undo_retention is over. If before-images, which are needed for a flashback query or flashback transaction query are missing, an ORA-1555 snapshot too old will occur.

This is how it was in ORACLE 9i. In Oracle 10g we now can have an undo tablespace with guaranteed undo_rentention. If such an undo tablespace runs full, the application will stop because no further before-images can be saved in the undo segments. So we can now make it directive that the server will have to keep all the before images needed. It is not just a request anymore.
The implementation of undo_retention was modified in 10g R1 and again in 10g R2. Oracle can now auto tune undo_retention, which is another step towards a self-tuning database. This parameter will become a hidden _parameter in future releases probably.

Starting with Oracle 10g a number of new flashback extensions have been implemented such as:
flashback versions query
flashback transaction query
flashback database
flashback drop
flashback table

Let’s take a look at them one by one.

Flashback versions query is an extension of flashback query. Now we can look at all the different versions, which a particular row had over a period of time in the past defined by a pair of SCNs ore timestamps:

SELECT employee_id, salary, versions_xid
FROM hr.employees
VERSIONS BETWEEN SCN MINVALUE AND SCN MAXVALUE
WHERE department_id=90;

This query will return all the versions of the salaries in the employees table, which are still available. The versions_xid pseudocolumn returns the ID of the transaction.

Once we have retrieved the ID of a transaction, we can now use it in order to find out what else this particular transaction has changed, by using the Static Data Dictionary View flashback transaction query:

desc flashback_transaction_query

XID RAW(8) Transaction identifier
START_SCN NUMBER Transaction start system change number (SCN)
START_TIMESTAMP DATE Transaction start timestamp
COMMIT_SCN NUMBER Transaction commit system change number (null for active transactions)
COMMIT_TIMESTAMP DATE Transaction commit timestamp (null for active transactions)
LOGON_USER VARCHAR2(30) Logon user for the transaction
UNDO_CHANGE NUMBER Undo system change number (1 or higher)
OPERATION VARCHAR2(32)
TABLE_NAME VARCHAR2(256)
TABLE_OWNER VARCHAR2(32)
ROW_ID VARCHAR2(19)
UNDO_SQL VARCHAR2(4000)

SELECT ””||rowid||”” AS king_rowid
FROM hr.employees
WHERE employee_id=100;

SELECT xid,
commit_scn,
TO_CHAR(commit_timestamp,
‘yyyymmddhh24mihhsscc’)
AS commit_timestamp,
operation
FROM flashback_transaction_query
WHERE table_name = ‘EMPLOYEES’
AND table_owner = ‘HR’
AND row_id =& king_rowid
ORDER BY commit_timestamp;

This will return all the versions of a single row and the transaction IDs. It can be useful for instance in a multi developer environment to find out who changed what and what else was change by a specific transaction:

SELECT *
FROM flashback_transaction_query
WHERE xid = HEXTORAW (‘&XID’);

To query the view flashback_transaction_query you need the select any transaction system privilege

Flashback versions query and flashback transaction query again use undo data.

Another new flashback technology was introduced in Oracle 10g release 1 to extend the functionality of flashback at table level. It is called flashback table and it uses a mechanism called row movement, which was introduced for partitioned tables already in Oracle 8i to permit updating a partition key and let the row move from partition to partition.
With row movement enabled for a table in Oracle 10g we can flashback the table to a point in time or SCN in the past. Row movement just gives the permission to Oracle to change the ROWID of a row. A ROWID is assigned on commit and is usually immutable.
If you find out that you have chosen the wrong point in time you can simply flashback the table again. Here is how it works:

ALTER TABLE hr.employees ENABLE ROW MOVEMENT;
FLASHBACK TABLE hr.employees TO TIMESTAMP to_timestamp(‘2005-27-10:19:00:00’, ‘YYYY-DD-MM:HH24:MI:SS’).

A flashback table to a specific SCN is also possible.

Flashback Database again is something completely different from all the other flashback functionalities from an implementation point of view. It enables the “roll back” of a complete database, including the data dictionary.

You first have to enable flashback logging before you can use flashback database. This is done by an entry in the controlfile (ALTER DATABASE FLASHBACK ON ;) . The database needs to be running in ARCHIVELOG mode.
There are three relevant parameters for flashback database. All of them are dynamically changeable:
recovery_file_dest (location of the Flash recovery area)
recovery_file_dest_size (size of the Flash recovery area)
db_flashback_retention_target (time in minutes how far you can flashback the database)

The technology used behind the scenes are so called flashback logs which are written from the buffer cache to the new flashback buffer (max. 16MB) in the SGA. From there the also new background process CTWR copies them to a file which you can imagine as some kind of giant rollback segment in the flash recovery area on disk. This area can optionally be used by multiple databases and does not only contain the flashback logs but can be used for all other files needed for backup and recovery as well, such as archived redologs, backupsets and image copies. In this area Oracle creates a structure of subfolders for every database using this flash recovery area and for the different kinds of files stored in it including the flashback logs.
These flashback logs are just images of changed blocks stored on disk. Once the server starts generating them you can ‘rollback’ the entire database to a point in time or an SCN in the past. Like a rewind button of a tape-recorder. The amount of flashback logs that needs to be preserved on disk is determined by the initialization parameter flashback_retention, as well as the amount of DMLs happening on the system.

Now a TRUNCATE TABLE by mistake is not a case for a full restore and a point in time recovery any more. We simply ‘rewind’ the database to before the truncate statement. The server will, in a first step flashback the database to a point in time short before the truncate by using the flashback logs for restore. And then, as a second step, recover the database forward to the desired point in time or SCN by applying redo from archived redo logs. We can open the database READ ONLY first in order to check if the flashback database was successful. We can try again multiple times if the point in time we have flashbacked to was the wrong one.
The flash recovery area, which holds the flashback logs is self-maintaining. The system will delete flashback logs as soon as they are not needed any more.
Oracle recommends the use of this new disk area to store all information needed for backup and recovery, such as flashback logs, archivelogs, image copies and backupsets.
A flashback database statement will fail if there is not enough flashback logs to rollback the database to the desired point in time in the past.

Flashback Drop uses an entirely different technology:
We now have a “recycle bin” in Oracle where a dropped table ends up.
We can restore a dropped table from the recycle bin and we even can query a dropped table from within the recycle bin.
How does this work?
When a table is dropped in 10g the space it uses is freed for reuse by other segments. You can in fact see the freed extents of the dropped table in DBA_FREE_SPACE.
However, Oracle will try to not overwrite this space as long as possible. And as long as this has not happened yet, we can read from the table residing in the recycle bin.
We can do so by using the table’s new name. It has just been renamed internally and we now have to use the new name to query it. We can see which objects are in the recycle bin by querying DBA_RECYCLEBIN.
A DROP TABLE … PURGE statement bypasses the recycle bin and the table will be gone straight away.
This new drop-behavior can be switched off in Oracle 10g R2 in order to get the same behavior as we had in former releases by setting the initialization parameter recyclebin to off. It was a hidden _parameter in Oracle 10gR1.

Starting with Oracle 10g release 2 we can even flashback the database to a previous incarnation which means we can rollback the system to an SCN which was created before the last open resetlogs. A flashback database in Oracle 10g R1was restricted to the same incarnation of the database.

Additionally we now have the option to create so-called named restore points and even guaranteed named restore points. Like savepoints in a transaction named restore points can be used as “saved points” for the entire database to which you can go back to any time. This is a handy alternative to the use of SCNs or timestamps.
Again flashback logs are used. This functionality works even without having enabled flashback logging enabled.
Flashing back a database can save recovery time dramatically because in fact there is no need for a full restore like before Oracle 10g with an incomplete recovery. On the other hand flashback logging has a drawback, which Oracle describes as a minor one: its cost in performance is in the single digit range.

All together we have a lot of different functionalities now, which can be useful to repair logical errors. Now it is ‘just’ a matter of having enough space available on disk to store all the information need for flashback.

For further information please do not hesitate to contact me by email:

lutz.hartmann@oracle.com
Senior Instructor Oracle Education

Baden, Switzerland
November 2005


Oracle Education Special Events this Year

January 21, 2006

Oracle Education Switzerland presents a number of special workshops in 2006:
For details and booking on all events please contact edu-buch_ch@oracle.com

May 2006:
The Tuning Expert Jonathan Lewis, who has just published his latest book about the Cost Based Optimizer will deliver a two day workshop in Zurich on May 11th. and 12th 2006.
There will be two blocks of seminar sessions:
- on the first day Jonathan Lewis will deal with indexing strategies
- on the second day he will talk about troubleshooting and tuning concepts.

This workshop is an absolute must for all developers and DBAs who want to get an into depth insight into the funtioning of the oracle machine.

September 2006:
There are the first people booking for Tom Kyte in Zurich in September 2006 already!!! After the very successful workshop in December 2005 Tom Kyte will come over here and deliver trainings on Sept. 21st. -22nd. 2006 again because there is such a big demand coming from the audience. It will be an event similar to the one in December 2005, but only two days this time.


deferred constraints part 2

January 13, 2006

What you have to be aware of when you create a deferrable primary key:

- the system will need an index for the primary key as you know
- if there already is an index for the column you want to add a primary key constraint to, then
- the server will check, if the index is usable state
- it will check if the index is a unique index (does not allow duplicate values)
- if the index is a unique index the server will check if the primary key you want ot add is deferrable
- if the existing index is unique and the primary key you want to add is not deferrable ( which is the default , remember),
then the existing index can be used for the primary key constraint
- if the existing is unique but the primary key you create is deferrable, the existing index cannot be used and needs to be dropped first, before a non unique index is created (an index which allows duplicate values).

Both, the dropping and the creating of the index might significantly consume time on the system!!
You should thing deep enough before you create a deferrable primary key. Maybe it is sufficiently enough for your application that the foreing key is deferrable and you do not need to create a deferrable primary key and so can use an already existing index.


deferred constraints part 1

January 12, 2006

Some interesting facts about constraints:
If you must update the primary key and the foreign key
and they are located in different tables then you can get into troubles:
If yo disable the foreign key it can happen that you will not be able to enable tit anymore because somebody has messed around with the values while it was disabled. You could get a full table lock to prevent this but this might be difficult also.

Today we have had a closer look at deferrable constraints.
These can be a good alternative for disabeling constraints for a cascading UPDATE.

It is essential to create the constraint as DEFERRABLE right away when creating it, if you want to be able to deferre the constraint chaeck to the end of the transaction.
This is not modifyable afterwards.
If you do not specify DEFERRABLE INITIALLY DEFEERED as creation time but only specify DEFERRABLE ,
then DEFERRABLE INITIALLY IMMEDIATE is the default.

Let’s see how they work:
PREREQUISITS:
- first I create a Primary key and a foreign key relationship.

SQL> ALTER TABLE dept ADD CONSTRAINT dept_pk PRIMARY KEY (department_id);
Table altered.

SQL> ALTER TABLE emp ADD CONSTRAINT emp_fk FOREIGN KEY (department_id)
2 REFERENCES DEPT (department_id)
3 DEFERRABLE;
Table altered.

SQL> SELECT constraint_name, search_condition, constraint_type, deferrable, deferred
2 FROM user_constraints
3* WHERE table_name=’EMP’

CONSTRAINT_NAME SEARCH_CONDITION C DEFERRABLE DEFERRED
—————————— —————————— – ————– ———
SYS_C002768 “LAST_NAME” IS NOT NULL C NOT DEFERRABLE IMMEDIATE
SYS_C002769 “EMAIL” IS NOT NULL C NOT DEFERRABLE IMMEDIATE
SYS_C002770 “HIRE_DATE” IS NOT NULL C NOT DEFERRABLE IMMEDIATE
SYS_C002771 “JOB_ID” IS NOT NULL C NOT DEFERRABLE IMMEDIATE
EMP_FK R DEFERRABLE IMMEDIATE
EMP_PK P NOT DEFERRABLE IMMEDIATE
EMP_FK_MGR R NOT DEFERRABLE IMMEDIATE
CHECK_SAL “SAL”>500 C NOT DEFERRABLE IMMEDIATE

DEMO 1:
- then I try my updates on the key and a foreign key, does not work of course
- now I set the deferrable constraints for my session to deferred
- then I try my updates on the key and a foreign key again and it works,
because the constraint-checking is deferred to the end of the transaction
- now I make a critical update which will cause a constraint volation at commit time
this will cause a TARNSACTION LEVEL ROLLBACK at commit time!!!!

DEMO 2:
- I do the same steps as in DEMO 1, only that I change the state of my deferrable constraints to immediate checking before my critical update.
=> this causes a STATEMENT LEVEL ROLLBACK right after the statement is processed,
the other two updates if issued before this are not rolled back this time!!!!!!

This is DEMO 1:

SQL> UPDATE emp SET department_id=department_id+1000;
UPDATE emp SET department_id=department_id+1000
*
ERROR at line 1:
ORA-02291: integrity constraint (HR.EMP_FK) violated – parent key not found

SQL> UPDATE dept SET department_id=department_id+1000;
UPDATE dept SET department_id=department_id+1000
*
ERROR at line 1:
ORA-02292: integrity constraint (HR.EMP_FK) violated – child record found

SQL> alter session set constraints=deferred;

SQL> UPDATE emp SET department_id=department_id+1000;
107 rows updated.

SQL> update dept set department_id=department_id+1000;
27 rows updated.

SQL> update dept set department_id=8888 where department_id=1090; # this is the critical update
1 row updated.

SQL> commit;
commit
*
ERROR at line 1:
ORA-02091: transaction rolled back
ORA-02292: integrity constraint (HR.EMP_FK) violated – child record found

This is DEMO 2:

SQL> update dept set department_id=department_id+1000;

27 rows updated.

SQL> update emp set department_id=department_id+1000;

107 rows updated.

SQL> alter session set constraints=immediate;

Session altered.

SQL> update dept set department_id=8888 where department_id=1090;
update dept set department_id=8888 where department_id=1090
*
ERROR at line 1:
ORA-02292: integrity constraint (HR.EMP_FK) violated – child record found

Some very good examples for the usage of deferred constraints you can find at asktom.oracle.com


Oracle Block Size Specials

January 11, 2006

Did you know that Linux x86 and Sun Solaris
are not capable of Blocks with sizes larger than 16k?

This is what happened to me yesterday on an Linux box with Red Hat Advanced Server 3:

Connected to:
Oracle9i Enterprise Edition Release 9.2.0.4.0 – Production
With the Partitioning, Oracle Label Security, OLAP and Oracle Data Mining option s
JServer Release 9.2.0.4.0 – Production


SQL> alter system set db_32k_cache_size = 32M;

alter system set db_32k_cache_size = 32M
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-00382: 32768 not a valid block size, valid range [2048..16384]

And here is what Metalink says:

Doc ID: Note:138242.1
Subject: 9i: ORA-382 Creating 32K Blocksize Tablespace and Related Cache Not Set
Type: PROBLEM
Status: PUBLISHED
This limitation is not an Oracle9i restriction, but is operating system -specific. Refer to the platform specifications for your operating system.

Oracle9i Administrator’s Reference     Release 1 (9.0.1) for UNIX Systems:

AIX-Based Systems,Compaq Tru64 UNIX,HP 9000 Series HP-UX,Linux Intel,Sun SPARC Solaris:

DB_BLOCK_SIZE:2048 to 16384 (Linux, Solaris)2048 to 32768 (AIX, HP, Tru64)

Oracle9i Database Administrator's Guide for Windows:

DB_BLOCK_SIZE: 2048 to 16384(Windows NT, 95, 98, 2000)

Have you ever created a tablespace
with a blocksize of -1 bytes on an Oracle database?
I had this recently with Oracle 9.2.0.4
(this was a patch with optimization potential I guess, hope it was fixed).
Is this a black hole, or is it some kind of anti-data you can store there?