ORA-01031: insufficient privileges with "as sysman"

This error "ORA-01031: insufficient privileges" really made me frustrated while trying to upgrade 11.2.0.2 to 11.2.0.3.

Basically, as an upgrade process it has to write some information to 11.2.0.2 ASM so it shutsdown ASM instance running on 11.2.0.2 and then try's to start up with 11.2.0.3 binaries and thats when I kept getting this error.

Following is the Error that I was getting at the time of migration:
--------------------------------------------------------------------------------------------

CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in inittab
Start of resource "ora.asm" failed
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'node1'
CRS-2676: Start of 'ora.drivers.acfs' on 'node11' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'node1'
ORA-01031: insufficient privileges
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-01031: insufficient privileges
. For details refer to "(:CLSN00107:)" in "/opt/grid/app/11.2.0.3/grid/log/node1/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.asm' on 'node1' failed
CRS-2679: Attempting to clean 'ora.asm' on 'node1'
ORA-01031: insufficient privileges

--------------------------------------------------------------------------------------------

So, tried to just start "sqlplus / as sysasm" (note with / as sysdba has no issues)

NONE::node1:/opt/oracle>export ORACLE_SID=+ASM1
+ASM1::node1:/opt/oracle>export ORACLE_HOME=/opt/grid/app/11.2.0.3/grid
+ASM1::node1:/opt/oracle>export PATH=$ORACLE_HOME/bin:$PATH

+ASM1::node1:/opt/oracle>sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Thu Apr 4 11:24:22 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

ERROR:
ORA-01031: insufficient privileges


First thought it might be the environment variables but not in my case.
ORACLE_HOME is set to 11.2.0.3 and PATH has $ORACLE_HOME/bin;$PATH but still gets the same error.

No issue when I set ORACLE_HOME back to 11.2.0.2 !!!! So, whats the difference?

Took me a while to figure this out but nailed it:
Basically, when I was doing upgrade OUI offers with "Privileged Operating Systems Groups" for ASM DBA, ASM Operator and ASM Administration.
I chose the default values for this and un-fortunately in my environment grid user "oracle" does not belong to a group named "asmadmin" and thus I get "insufficient privileges"

How do I change it?

Check the file config.c under /opt/grid/app/11.2.0.3/grid/rdbms/lib/

You will see the following:


#define SS_DBA_GRP "asmdba"
#define SS_OPER_GRP "asmoper"
#define SS_ASM_GRP "asmadmin" --> this was my problem. I changed this to "dba".

Its not done yet.

We need to recompile oracle binary to get this change affected.

Here is how you do that:

node1:/opt/grid/app/11.2.0.3/grid/rdbms/lib>make -f ins_rdbms.mk config.o ioracle

Make sure to check the output thoroughly as you might see errors in 11g environment as the folder/files permissions might be owned by "root" and it might fail to move the old oracle file to oracleO and copy the new file.
If you do see such errors then ask your admin to replace the old oracle file from /opt/grid/app/11.2.0.3/grid/bin/oracle to /opt/grid/app/11.2.0.3/grid/bin/oracleO and copy the file from
/opt/grid/app/11.2.0.3/grid/rdbms/lib/oracle to /opt/grid/app/11.2.0.3/grid/bin/

That is it:
The error vanishes now.


+ASM1:node1:/opt/grid/app/11.2.0.3/grid/rdbms/lib>sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Thu Apr 4 17:12:55 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

Good luck. I am having another migration issue right now and will see what else is messed up.