Friday, October 5, 2012

Data Guard Errors: ORA-00308 ORA-17503 ORA-15012

Error
DataGuard redo apply is broken. No more logs are applied. Alert Log of Standby shows following error.
This happened after server was crashed.
Errors with log +PSSFLASH/PRODSBY/archivelog/2012_10_04/thread_3_seq_19610.689.795807751
Errors in file /ora/app/oracle/diag/rdbms/PRODSBY/PRODSBY2/trace/PRODSBY2_pr00_18575.trc:
ORA-00308: cannot open archived log '+PSSFLASH/PRODSBY/archivelog/2012_10_04/thread_3_seq_19610.689.795807751'
ORA-17503: ksfdopn:2 Failed to open file +PSSFLASH/PRODSBY/archivelog/2012_10_04/thread_3_seq_19610.689.795807751

Missing File
As per alertlog below is missing file
+PSSFLASH/PRODSBY/archivelog/2012_10_04/thread_3_seq_19610.689.795807751

Production (PROD)
Login to asmcmd and copy file from prod(PROD) to filesystem
asmcmd -p
cd +PSSFLASH/PROD/ARCHIVELOG/2012_10_04
cp thread_3_seq_19610.2065.795819191 /tmp/thread_3_seq_19610.2065.795819191

Standby Side (PRODSBY)
Stop Recovery if any in progress [ It would have already broken with error ]
SQL > ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
scp <primaryhost>:/tmp/thread_3_seq_19610.2065.795819191 /prodstage01/keepsafe
SQL> ALTER DATABASE REGISTER OR REPLACE PHYSICAL LOGFILE '/prodstage01/keepsafe/thread_3_seq_19610.2065.795819191';
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE;

Problem is fixed
Standby Starts Media recovery and will soon catchup