RMAN-20011: target database incarnation is not current in recovery catalog

A surprising database backup failure report was delivered to my mailbox, upon checking with the log revealed the below pasted RMAN- errors. This left me wondering as to how come the rman backup would fail for no reason with an error like this, there is something fishy about the situation.

RMAN>
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-06004: ORACLE error from recovery catalog database: RMAN-20011: target 
database incarnation is not current in recovery catalog

here are the steps below that were performed to fix the issue of incarnation.

UNIX:~/dba/log > rman target / catalog rman_user/rman_pass@rcat.acme.com

Recovery Manager: Release 10.2.0.1.0 - Production on Mon Jan 26 06:45:20 2009

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database: PROD (DBID=2706003900)
connected to recovery catalog database

RMAN>  list incarnation of database;


RMAN> exit

as you can see above, no incarnations listed.

UNIX:~/dba/log >  . oraenv
ORACLE_SID = [oracle] ? PROD
UNIX:~/dba/log > rman target / nocatalog

Recovery Manager: Release 10.2.0.1.0 - Production on Mon Jan 26 06:46:58 2009

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database: PROD (DBID=2706003900)
using target database control file instead of recovery catalog

RMAN> list incarnation of database;


List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       PROD 2706003900       CURRENT 1          07-FEB-07

RMAN> exit

Now, having logged into the recovery catalog database as the recovery catalog owner and running a couple of queries shown below will give us some background information on the target database DBID and the incarnation id etc.

SQL> select dbinc_key,db_key,db_name from RMAN.dbinc where db_name='PROD';

 DBINC_KEY     DB_KEY DB_NAME  
---------- ---------- -------- 
    875312     875311 PROD     


SQL> select DBID, NAME, DBINC_KEY, RESETLOGS_CHANGE#, RESETLOGS_TIME 
  2  from rc_database_incarnation where dbid=2706003900;

      DBID NAME      DBINC_KEY RESETLOGS_CHANGE# RESETLOGS
---------- -------- ---------- ----------------- ---------
2706003900 PROD     875312                     1 07-FEB-07
2706003900 PROD2    9179422           6.2038E+12 24-MAY-09

SQL> alter session set nls_date_format = 'DD-MON-YYYY HH24:MI:SS' ;

Session altered.

SQL>  select DBID, NAME, DBINC_KEY, RESETLOGS_CHANGE#, RESETLOGS_TIME
  2   from rc_database_incarnation where dbid=2706003900;

      DBID NAME      DBINC_KEY RESETLOGS_CHANGE# RESETLOGS_TIME
---------- -------- ---------- ----------------- --------------------
2706003900 PROD      875312                    1 07-FEB-2007 15:49:16
2706003900 PROD2    9179422           6.2038E+12 24-MAY-2009 22:25:13

The database PROD with the DBID (2706003900) and DBINC_KEY (875312) has to be reset in the recovery catalog and re-registered for its backups to resume.

UNIX:~/dba/log > rman target / catalog rman_user/rman_pass@rcat.acme.com

Recovery Manager: Release 10.2.0.1.0 - Production on Mon Jan 26 07:00:56 2009

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database: PROD (DBID=2706003900)
connected to recovery catalog database

RMAN> RESET DATABASE TO INCARNATION 875312;

database reset to incarnation 875312

RMAN> register database;

starting full resync of recovery catalog


full resync complete
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of register command on default channel at 05/26/2009 07:40:05
RMAN-20002: target database already registered in recovery catalog

RMAN>
RMAN>

It should have rather been a "resync catalog" than "register database" to suppress the above error.After this, the backup ran to success.

RMAN> list incarnation of database;


List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN    Reset Time
------- ------- -------- ---------------- ---     ----------   ----------
875311  875312  PROD     2706003900       CURRENT 1             07-FEB-07
875311  9179422 PROD2    2706003900       ORPHAN  6203845546338 24-MAY-09

RMAN> exit


Recovery Manager complete.