Ora-27303: additional information: startup egid = 80, current egid = 82

While using DBUA to upgrade database from 11.2.0.2 to 11.2.0.3, DBUA was throwing an error starting with PRCR-1079: Failed to start resource ora.xxx.db followed by bunch of other errors.

One error in between those error stack kinda made me obvious to look the file "oracle" permissions.

Ora-27303: additional information: startup egid = 80 (dba), current egid = 82 (asmadmin)

So, comparing the file permissions on both Oracle Home and Grid Home tells me that they are not same:

/opt/oracle>ls -al app/product/11.2.0.3/db_1/bin/oracle

-rwsr-s--x 1 oracle dba 232399431 Apr 29 12:39 app/product/11.2.0.3/db_1/bin/oracle

/opt/oracle>

/opt/oracle>ls -al /opt/grid/app/11.2.0.3/grid/bin/oracle

-rwxr-xr-x 1 oracle dba 203973009 Apr  4 17:08 /opt/grid/app/11.2.0.3/grid/bin/oracle

/opt/oracle>

Changing the permissions on Grid Home Oracle and after this DBUA had no issues.

/opt/oracle>cd /opt/grid/app/11.2.0.3/grid/bin/

/opt/grid/app/11.2.0.3/grid/bin>chmod 6751 oracle

/opt/grid/app/11.2.0.3/grid/bin>ls -al oracle

-rwsr-s--x 1 oracle dba 203973009 Apr  4 17:08 oracle

/opt/grid/app/11.2.0.3/grid/bin>

ORA-01031: insufficient privileges with "as sysman"

This error "ORA-01031: insufficient privileges" really made me frustrated while trying to upgrade 11.2.0.2 to 11.2.0.3.

Basically, as an upgrade process it has to write some information to 11.2.0.2 ASM so it shutsdown ASM instance running on 11.2.0.2 and then try's to start up with 11.2.0.3 binaries and thats when I kept getting this error.

Following is the Error that I was getting at the time of migration:
--------------------------------------------------------------------------------------------

CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in inittab
Start of resource "ora.asm" failed
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'node1'
CRS-2676: Start of 'ora.drivers.acfs' on 'node11' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'node1'
ORA-01031: insufficient privileges
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-01031: insufficient privileges
. For details refer to "(:CLSN00107:)" in "/opt/grid/app/11.2.0.3/grid/log/node1/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.asm' on 'node1' failed
CRS-2679: Attempting to clean 'ora.asm' on 'node1'
ORA-01031: insufficient privileges

--------------------------------------------------------------------------------------------

So, tried to just start "sqlplus / as sysasm" (note with / as sysdba has no issues)

NONE::node1:/opt/oracle>export ORACLE_SID=+ASM1
+ASM1::node1:/opt/oracle>export ORACLE_HOME=/opt/grid/app/11.2.0.3/grid
+ASM1::node1:/opt/oracle>export PATH=$ORACLE_HOME/bin:$PATH

+ASM1::node1:/opt/oracle>sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Thu Apr 4 11:24:22 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

ERROR:
ORA-01031: insufficient privileges


First thought it might be the environment variables but not in my case.
ORACLE_HOME is set to 11.2.0.3 and PATH has $ORACLE_HOME/bin;$PATH but still gets the same error.

No issue when I set ORACLE_HOME back to 11.2.0.2 !!!! So, whats the difference?

Took me a while to figure this out but nailed it:
Basically, when I was doing upgrade OUI offers with "Privileged Operating Systems Groups" for ASM DBA, ASM Operator and ASM Administration.
I chose the default values for this and un-fortunately in my environment grid user "oracle" does not belong to a group named "asmadmin" and thus I get "insufficient privileges"

How do I change it?

Check the file config.c under /opt/grid/app/11.2.0.3/grid/rdbms/lib/

You will see the following:


#define SS_DBA_GRP "asmdba"
#define SS_OPER_GRP "asmoper"
#define SS_ASM_GRP "asmadmin" --> this was my problem. I changed this to "dba".

Its not done yet.

We need to recompile oracle binary to get this change affected.

Here is how you do that:

node1:/opt/grid/app/11.2.0.3/grid/rdbms/lib>make -f ins_rdbms.mk config.o ioracle

Make sure to check the output thoroughly as you might see errors in 11g environment as the folder/files permissions might be owned by "root" and it might fail to move the old oracle file to oracleO and copy the new file.
If you do see such errors then ask your admin to replace the old oracle file from /opt/grid/app/11.2.0.3/grid/bin/oracle to /opt/grid/app/11.2.0.3/grid/bin/oracleO and copy the file from
/opt/grid/app/11.2.0.3/grid/rdbms/lib/oracle to /opt/grid/app/11.2.0.3/grid/bin/

That is it:
The error vanishes now.


+ASM1:node1:/opt/grid/app/11.2.0.3/grid/rdbms/lib>sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Thu Apr 4 17:12:55 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

Good luck. I am having another migration issue right now and will see what else is messed up.

"_allow_resetlogs_corruption" recovery....

Please DO NOT use this in any production environment without Oracle Support...

*** Using this parameter should be your LAST option as you do not have any proper backups to bring your database online ***

One of our database was kind of un-opened mode due to system tablespace datafile corruption (for any reason like restored from old backup etc.,)

We are OK to loose some data and some tablespaces which were created later point than the SCN that system tablespace has. So, used the parameter "_allow_resetlogs_corruption" an un-documented (repeat Oracle will not support your database if you recover your database with this option without their support) parameter. Below these errors shows the use of this parameter and I was able to recover database and open.

Database mounted.
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01245: offline file 1 will be lost if RESETLOGS is done
ORA-01110: data file 1: '/oradata/db_name/datafile//system.282.757162799'

SQL> alter database open noresetlogs;
alter database open noresetlogs
*
ERROR at line 1:
ORA-01588: must use RESETLOGS option for database open

SQL> alter database datafile 1 online;

Database altered.

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1: '/oradata/db_name/datafile/system.282.757162799'

Use of the parameter:


SQL> alter system set "_allow_resetlogs_corruption"=true scope=spfile;

System altered.

SQL> shutdown
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area 1469792256 bytes
Fixed Size                  2121312 bytes
Variable Size            1107296672 bytes
Database Buffers          352321536 bytes
Redo Buffers                8052736 bytes
Database mounted.
SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-01610: recovery using the BACKUP CONTROLFILE option must be done

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01248: file 19 was created in the future of incomplete recovery
ORA-01110: data file 19: '/oradata/db_name/datafile/file2.dbf'

I know for sure I do not need to worry about this datafile as I have data that I can put back in so marking it as OFFLINE.

SQL> alter database datafile 19 offline;

Database altered.

Now Open with resetlogs and you should be good. Take a FULL Backup ASAP.

SQL> alter database open resetlogs;

Database altered.

SQL>

Once again, Please do not use this parameter in any production or critical databases without Oracle Support and recommendation otherwise you will end up with inconsistency database.