Wednesday, November 18, 2009

spfile in 9i, 10g and 11g

There is a quite a difference in how your ouput shows up in 9i, 10g and 11g when you do a "show parameter spfile", here is how it shows up if the instance was brought up using an spfile.




Oracle 9i

SQL> sho parameter spfile

NAME TYPE VALUE
-------------------------- ----------- ------------------------------
spfile string ?/dbs/spfile@.ora

Oracle 10g

SQL> sho parameter spfile

NAME TYPE VALUE
-------------------------- ----------- ------------------------------
spfile string /oracle/product/10.2.0/dbs/spfile10G.ora

Oracle 11g

SQL> sho parameter spfile

NAME TYPE VALUE
-------------------------- ----------- ------------------------------
spfile string /oracle/product/11.1.0/dbs/spfile11G.ora




The output in 9i does not show the full ORACLE_HOME location, neither does it display the spfile name while in 10g and above the full ORACLE_HOME location gets displayed along with the spfile name.

Monday, November 9, 2009

ORA-00600 during controlfile recreation

The destructive errors like ORA-00600, ORA-07445 etc show up at unusual times, one such perfect example of getting an ORA-00600 is explained below - it appeared during the execution of CREATE CONTROLFILE script. Before we go any further, here is a quick background of what's being done. A 10.2.0.1 database is to be migrated from one server to another.

As a proactive DBA, which I am not most of the time ;-) - I dumped the control file to the trace and went about creating the scripts to copy the files from the source to destination. All went well until I confronted ORA-600 during the control file recreation.

I had to quickly check the Oracle version which i am currently using, just to make sure there is not a mis-match between the source and destination Oracle binaries. The error pointed me out to a trace file that Oracle usually generates when it needs to provide more information on a particular error.




SQL> CREATE CONTROLFILE REUSE DATABASE "PROD" NORESETLOGS ARCHIVELOG
*
ERROR at line 1:
ORA-01503: CREATE CONTROLFILE failed
ORA-00600: internal error code, arguments: [kccscf_1], [9], [106012], [65535],
[], [], [], []






here is the error entry in alert log






Errors in file /prd/u01/PROD/udump/prod_ora_12326.trc:
ORA-00600: internal error code, arguments: [kccscf_1], [9], [106012], [65535], [], [], [], []
ORA-1503 signalled during: CREATE CONTROLFILE REUSE DATABASE "PROD" NORESETLOGS ARCHIVELOG


UNIX$> more /prd/u01/PROD/udump/prod_ora_12326.trc

ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [kccscf_1], [9], [106012], [65535], [], [], [], []
Current SQL statement for this session:
CREATE CONTROLFILE REUSE DATABASE "PROD" NORESETLOGS ARCHIVELOG
MAXLOGFILES 255
MAXLOGMEMBERS 5
MAXDATAFILES 12144
MAXINSTANCES 12
MAXLOGHISTORY 106012






Everything looked fine but there is no clue as to what's causing the mighty ORA-600 in this occasion, I gave it a couple of tries myself to troubleshoot before turning over to Mr. Ellison's boys for help!!!. A severity - 1 was raised for this issue and I was pointed out at the value of MAXLOGHISTORY to be equal to 65535 as the maximum limit against the value that I had as 106012.

This looks silly, how come a trace controlfile that was dumped from an SqlPlus session have a value not recommended by Oracle itself. It was later reported by the Oracle technician that its a Bug in 10.2.0.1 which has been fixed in 10.2.0.4. Corrected script is shown below.







CREATE CONTROLFILE REUSE DATABASE "PROD" NORESETLOGS ARCHIVELOG
MAXLOGFILES 255
MAXLOGMEMBERS 5
MAXDATAFILES 12144
MAXINSTANCES 12
MAXLOGHISTORY 65535


Wednesday, November 4, 2009

Datafile naming error

A text editor formatting flaw could really end a life of a DBA in jeopardy, that is what I felt when I realized that one of the tablespaces that I created had a filename with white spaces - such a database gone live into the production is a nightmarish experience. Here is the situation, a tablespace creation script (vi editor) had a filename broken down to the next line and I thought it just came down to the next line after having reached the end of line, but it wasn't. It looked like shown below



CREATE TABLESPACE "USERS_BIG"
LOGGING
DATAFILE '/prd/u01/oradata/prod/users_big01.dbf' SIZE 2500M REUSE,
'/prd/u01/oradata/prod/use
rs_big02.dbf' SIZE 2500M REUSE,
'/prd/u01/oradata/prod/users_big03.dbf' SIZE 5000M REUSE;



This is what it looks like from the inside, of the database and that of the Operating System.



SQL> select file_name from dba_data_files
2 where tablespace_name like '%BIG%';

FILE_NAME
------------------------------------------------------
/prd/u01/oradata/prod/users_big01.dbf
/prd/u01/oradata/prod/use
rs_big02.dbf

/prd/u01/oradata/prod/users_big03.dbf

UNIX:/prd/u01/oradata/prod > ls -ltr
total 184320194
-rw-r----- 1 oracle dba 5242888192 May 22 17:12 users02.dbf
-rw-r----- 1 oracle dba 2621448192 May 22 17:12 use
rs_big02.dbf
-rw-r----- 1 oracle dba 5242888192 May 22 17:12 users01.dbf
-rw-r----- 1 oracle dba 524296192 May 22 17:12 tools02.dbf
-rw-r----- 1 oracle dba 524296192 May 22 17:12 tools01.dbf



Here are some vain efforts to locate the file individually, but nothing worked out.



UNIX:/prd/u01/oradata/prod > ls -ltri
total 184320194
20 -rw-r----- 1 oracle dba 5242888192 May 22 17:12 users02.dbf
19 -rw-r----- 1 oracle dba 5242888192 May 22 17:12 users01.dbf
30 -rw-r----- 1 oracle dba 2621448192 May 22 17:12 use
rs_big02.dbf
18 -rw-r----- 1 oracle dba 524296192 May 22 17:12 tools02.dbf
17 -rw-r----- 1 oracle dba 524296192 May 22 17:12 tools01.dbf

UNIX:/prd/u01/oradata/prod > find ./ -inum "30" -print
./use
rs_big02.dbf
UNIX:/prd/u01/oradata/prod > find ./ -inum "30" -exec ls -ltr {} \;
-rw-r----- 1 oracle dba 2621448192 May 22 17:12 ./use
rs_big02.dbf
UNIX:/prd/u01/oradata/prod > find ./ -inum "30" -exec ls -ltr {} \;|awk '{print $9}'
./use
rs_big02.dbf

UNIX:/prd/u01/oradata/prod > cp "users_big02.dbf" tempos.dbf
cp: cannot access users_big02.dbf



I made a copy of the broken name datafile to a corrected name datafile at the OS level and then renamed it at the database level by including the whitespaces in the source datafile argument of the 'alter database rename file' statement. The copying at the OS level required the entire white spaces to be included in the source file name argument of the 'cp' command, while being enclosed in double quotes (") as shown below.



UNIX:/prd/u01/oradata/prod > cp "use
> rs_big02.dbf" users_big02.dbf

SQL> select file_name from dba_data_files
2 where tablespace_name like '%BIG%';

FILE_NAME
----------------------------------------------------
/prd/u01/oradata/prod/users_big01.dbf
/prd/u01/oradata/prod/use
rs_big02.dbf

/prd/u01/oradata/prod/users_big03.dbf
/prd/u01/oradata/prod/users_big04.dbf


SQL> select tablespace_name,status from dba_tablespaces;

TABLESPACE_NAME STATUS
------------------------------ ---------
SYSTEM ONLINE
UNDO ONLINE
SYSAUX ONLINE
TEMP ONLINE
INDX ONLINE
PERFSTAT ONLINE
TOOLS ONLINE
USERS ONLINE
USERS_BIG ONLINE

9 rows selected.


SQL> alter tablespace users_big offline normal;

Tablespace altered.

SQL> select tablespace_name,status from dba_tablespaces;
TABLESPACE_NAME STATUS
------------------------------ ---------
SYSTEM ONLINE
UNDO ONLINE
SYSAUX ONLINE
TEMP ONLINE
INDX ONLINE
PERFSTAT ONLINE
TOOLS ONLINE
USERS ONLINE
USERS_BIG OFFLINE

9 rows selected.

SQL> !ls -ltr
total 185090258

-rw-r----- 1 oracle dba 5242888192 May 23 04:22 users_big03.dbf
-rw-r----- 1 oracle dba 2621448192 May 23 04:22 users_big01.dbf
-rw-r----- 1 oracle dba 2621448192 May 23 04:22 use
rs_big02.dbf
-rw-r----- 1 oracle dba 10485768192 May 23 04:22 undo.dbf
-rw-r----- 1 oracle dba 209715712 May 23 04:22 redo_PROD_2a.log


SQL> select sum(bytes)/1024/1024,file_name from dba_data_files
2 where tablespace_name like '%USER%'
3 group by file_name order by file_name;

SUM(BYTES)/1024/1024 FILE_NAME
-------------------- -----------------------------------------------
/prd/u01/oradata/prod/use
rs_big02.dbf

5000 /prd/u01/oradata/prod/users01.dbf
5000 /prd/u01/oradata/prod/users02.dbf
5000 /prd/u01/oradata/prod/users03.dbf
5000 /prd/u01/oradata/prod/users04.dbf
/prd/u01/oradata/prod/users_big01.dbf
/prd/u01/oradata/prod/users_big03.dbf
/prd/u01/oradata/prod/users_big04.dbf

8 rows selected.


SQL> alter database rename file '/prd/u01/oradata/prod/use
2 rs_big02.dbf' to '/prd/u01/oradata/prod/users_big02.dbf';

Database altered.


SQL> select sum(bytes)/1024/1024,file_name from dba_data_files
2 where tablespace_name like '%USER%'group by file_name
3 order by file_name;

SUM(BYTES)/1024/1024 FILE_NAME
-------------------- ------------------------------------------------
5000 /prd/u01/oradata/prod/users01.dbf
5000 /prd/u01/oradata/prod/users02.dbf
5000 /prd/u01/oradata/prod/users03.dbf
5000 /prd/u01/oradata/prod/users04.dbf
/prd/u01/oradata/prod/users_big01.dbf
/prd/u01/oradata/prod/users_big02.dbf
/prd/u01/oradata/prod/users_big03.dbf
/prd/u01/oradata/prod/users_big04.dbf

8 rows selected.


SQL> select * from v$recover_file;

FILE# ONLINE ONLINE_ ERROR CHANGE# TIME
---------- ------- ------- ----------------- ---------- ---------
18 OFFLINE OFFLINE OFFLINE NORMAL 0
19 OFFLINE OFFLINE OFFLINE NORMAL 0
20 OFFLINE OFFLINE OFFLINE NORMAL 0
21 OFFLINE OFFLINE OFFLINE NORMAL 0

SQL> alter tablespace USERS_BIG online;

Tablespace altered.


SQL> select * from v$recover_file;

no rows selected


SQL> select sum(bytes)/1024/1024,file_name from dba_data_files
2 where tablespace_name like '%USER%'
3 group by file_name order by file_name;

SUM(BYTES)/1024/1024 FILE_NAME
-------------------- ----------------------------------------------------
5000 /prd/u01/oradata/prod/users01.dbf
5000 /prd/u01/oradata/prod/users02.dbf
5000 /prd/u01/oradata/prod/users03.dbf
5000 /prd/u01/oradata/prod/users04.dbf
2500 /prd/u01/oradata/prod/users_big01.dbf
2500 /prd/u01/oradata/prod/users_big02.dbf
5000 /prd/u01/oradata/prod/users_big03.dbf
5000 /prd/u01/oradata/prod/users_big04.dbf

8 rows selected.



Removing the file at the OS level is a tricky job, the safest is to locate the inode number of the file and use the 'inum' flag of the 'find' command to remove it with the 'exec' flag as shown below



UNIX:/prd/u01/oradata/prod > find ./ -inum "30" -print
./use
rs_big02.dbf
UNIX:/prd/u01/oradata/prod > find ./ -inum "30" -exec ls -ltr {} \;
-rw-r----- 1 oracle dba 2621448192 May 23 04:22 ./use
rs_big02.dbf
UNIX:/prd/u01/oradata/prod > find ./ -inum "30" -exec rm {} \;
UNIX:/prd/u01/oradata/prod >


UNIX:/prd/u01/oradata/prod > ls -tlr
total 184320194
-rw-r----- 1 oracle dba 1048584192 May 23 04:30 sysaux.dbf
-rw-r----- 1 oracle dba 10485768192 May 23 04:42 undo.dbf
-rw-r----- 1 oracle dba 5242888192 May 23 04:45 users_big04.dbf
-rw-r----- 1 oracle dba 5242888192 May 23 04:45 users_big03.dbf
-rw-r----- 1 oracle dba 2621448192 May 23 04:45 users_big02.dbf
-rw-r----- 1 oracle dba 2621448192 May 23 04:45 users_big01.dbf
-rw-r----- 1 oracle dba 9961480192 May 23 04:45 system01.dbf


SQL> select tablespace_name,status from dba_tablespaces;

TABLESPACE_NAME STATUS
------------------------------ ---------
SYSTEM ONLINE
UNDO ONLINE
SYSAUX ONLINE
TEMP ONLINE
INDX ONLINE
PERFSTAT ONLINE
TOOLS ONLINE
USERS ONLINE
USERS_BIG ONLINE

9 rows selected.