Monday, April 5, 2021

Migrate ASMLIB to ASMFD

# --------------------------------------------------------------------------------------------

# ASMFD : How to Migrate ASM Diskgroups from ASMLIB to ASMFD (ASM Filter Driver) on Oracle Grid Infrastructure (RAC) (Doc ID 2172754.1)


# --------------------------------------------------------------------------------------------

# Applies to:


Oracle Database Backup Service - Version N/A and later

Oracle Database Cloud Exadata Service - Version N/A and later

Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]

Oracle Database Cloud Schema Service - Version N/A and later

Oracle Database Exadata Express Cloud Service - Version N/A and later Linux x86-64


# --------------------------------------------------------------------------------------------

# What is Oracle ASM Filter Driver (Oracle ASMFD) ?


Oracle ASM Filter Driver (Oracle ASMFD) is a kernel module that resides in the I/O path of the Oracle ASM disks. Oracle ASM uses the filter driver to validate write I/O requests to Oracle ASM disks.

The Oracle ASMFD simplifies the configuration and management of disk devices by eliminating the need to rebind disk devices used with Oracle ASM each time the system is restarted.

The Oracle ASM Filter Driver rejects any I/O requests that are invalid. This action eliminates accidental overwrites of Oracle ASM disks that would cause corruption in the disks and files within the disk group. For example, the Oracle ASM Filter Driver filters out all non-Oracle I/Os which could cause accidental overwrites.

Oracle ASM Filter Driver (Oracle ASMFD) is installed with an Oracle Grid Infrastructure installation. If you have an existing Oracle ASM library driver (Oracle ASMLIB) configuration, then this document will guide you to migrate the existing ASM diskgroup disks which are stamped with ASMLib to ASMFD (ASM Filter Driver).


# --------------------------------------------------------------------------------------------

# Solution


Following assumptions are made before we proceed with the next steps 


1.  Please check the existing environment is certified or not. Check below document


ASMFD (ASM Filter Driver) Support on OS Platforms (Certification Matrix).(Doc ID 2034681.1)


2.  It is recommended to apply latest PSU for successful migration.


3.  Take the backup of the database before proceeding this activity.


# --------------------------------------------------------------------------------------------

# Step by Step Procedure


1.  As a root user run below command to generate the which ASM Disk is associated with the OS disk. In a multipath configuration, please use the multipath name


[ root@srv10:/home: ]$ oracleasm listdisks | xargs oracleasm querydisk -p

Disk "DATA" is a valid ASM disk /dev/sdb1: LABEL="DATA" TYPE="oracleasm"

Disk "FRA" is a valid ASM disk /dev/sdc1: LABEL="FRA" TYPE="oracleasm"


2.  As a Grid Infrastructure (GI) software owner, get the present ASM diskstring value

[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p dsget

parameter:/dev/oracleasm/disks/*

profile:/dev/oracleasm/disks/*

[ +ASM ]|[ oracle@srv10:/home/oracle: ]$


In case the ASMLIB disk discovery string is NOT /dev/oracleasm/disks/* but "ORCL:*".

grid@hostname1 ~]$ asmcmd dsget

parameter:ORCL:*

profile:ORCL:*


3.  As a GI owner update the Oracle ASM disk discovery string to enable Oracle ASMFD to discover devices in the future.


[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p dsset '/dev/oracleasm/disks/*','AFD:*'

[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p dsget

parameter:/dev/oracleasm/disks/*, AFD:*

profile:/dev/oracleasm/disks/*,AFD:*



In case the ASMLIB disk discovery string is NOT /dev/oracleasm/disks/* but "ORCL:*".


[grid@hostname1 ~]$ asmcmd dsset 'ORCL:*','AFD:*'

[grid@hostname1 ~]$ asmcmd dsget

parameter:ORCL:*, AFD:*

profile:ORCL:*,AFD:*



4.  As a root user stop the cluster accessing the shared ASMLib disks


# Cluster

[ root@srv10:/root: ]$ crsctl stop crs



# Standalone

[ root@srv10:/root: ]$ crsctl stop has

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'srv10'

CRS-2673: Attempting to stop 'ora.dbprod.db' on 'srv10'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'srv10'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'srv10' succeeded

CRS-2677: Stop of 'ora.dbprod.db' on 'srv10' succeeded

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'srv10'

CRS-2673: Attempting to stop 'ora.FRA.dg' on 'srv10'

CRS-2677: Stop of 'ora.DATA.dg' on 'srv10' succeeded

CRS-2677: Stop of 'ora.FRA.dg' on 'srv10' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'srv10'

CRS-2677: Stop of 'ora.asm' on 'srv10' succeeded

CRS-2673: Attempting to stop 'ora.evmd' on 'srv10'

CRS-2677: Stop of 'ora.evmd' on 'srv10' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'srv10'

CRS-2677: Stop of 'ora.cssd' on 'srv10' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'srv10' has completed

CRS-4133: Oracle High Availability Services has been stopped.


5.  Check and Stop the oracleasm driver and acfsload as root user


[ root@srv10:/root: ]$ oracleasm status

Checking if ASM is loaded: yes

Checking if /dev/oracleasm is mounted: yes


[ root@srv10:/root: ]$ oracleasm exit

Unmounting ASMlib driver filesystem: /dev/oracleasm

Unloading module "oracleasm": oracleasm


[ root@srv10:/root: ]$ ls -ltr /dev/oracleasm/

total 0


[ root@srv10:/root: ]$ lsmod | grep acfs

oracleacfs 3498241 2


oracleoks 503994 2 oracleacfs,oracleadvm


[ root@srv10:/root: ]$ acfsload stop


[ root@srv10:/root: ]$ lsmod | grep acfs


# --------------------------------------------------------------------------------------------

# Note: It is recommended to remove the ASMLib related RPMs to avoid any conflict.


[ root@srv10:/root: ]$ rpm -qa | grep asm

oracleasmlib-2.0.12-1.el7.x86_64

kmod-oracleasm-2.0.8-28.0.1.el7.x86_64

oracleasm-support-2.1.11-2.el7.x86_64


[ root@srv10:/root: ]$ rpm -e oracleasm-support-2.1.11-2.el7.x86_64 kmod-oracleasm-2.0.8-28.0.1.el7.x86_64 oracleasmlib-2.0.12-1.el7.x86_64

warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave


6.  As root, configure Oracle ASMFD to filter at the node level:


[ root@srv10:/root: ]$ afdroot version_check

AFD-616: Valid AFD distribution media detected at: '/u01/app/19.3.0.1/grid/usm/install/Oracle/EL7UEK/x86_64/5.4.17-2011.0.7/5.4.17-2011.0.7-x86_64/bin'


[ root@srv10:/root: ]$ afddriverstate supported

AFD-9200: Supported


[ root@srv10:/root: ]$ asmcmd afd_configure

AFD-627: AFD distribution files found.

AFD-634: Removing previous AFD installation.

AFD-635: Previous AFD components successfully removed.

AFD-9294: updating file /etc/sysconfig/oracledrivers.conf

AFD-636: Installing requested AFD software.

AFD-637: Loading installed AFD drivers.

AFD-9321: Creating udev for AFD.

AFD-9323: Creating module dependencies - this may take some time.

AFD-9154: Loading 'oracleafd.ko' driver.

AFD-649: Verifying AFD devices.

AFD-9156: Detecting control device '/dev/oracleafd/admin'.

AFD-638: AFD installation correctness verified.

Modifying resource dependencies - this may take some time.



7.  As the Oracle Grid Infrastructure / root owner verify the status of Oracle ASMFD:

[ root@srv10:/root: ]$ asmcmd afd_state

ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'srv10.dbnitro.net'


# Case is disabled

ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'DISABLED' on host 'hostname1'           <<<< Filtering is disabled


8.  As a root user set the AFD diskstring and enable the Filtering


[ root@srv10:/root: ]$ asmcmd -p afd_filter -e

[ root@srv10:/root: ]$ asmcmd -p afd_state

ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'srv10.dbnitro.net'      <<<< Filtering is enabled



9.  As a root user, with the help of the data gathered in step 1, migrate (label) all the ASMLib disks 

$ORACLE_HOME/bin/asmcmd afd_label <OLD ASMLib Disk Label Name> <Disk Path> --migrate


[ root@srv10:/root: ]$ cat /etc/oracleafd.conf

afd_diskstring='/dev/sd*'

afd_filtering=enable


[ root@srv10:/root: ]$ asmcmd afd_scan


[ root@srv10:/root: ]$ asmcmd afd_lsdsk


[ root@srv10:/root: ]$ asmcmd -p afd_label DATA /dev/sdb1 --migrate


[ root@srv10:/root: ]$ asmcmd -p afd_label FRA /dev/sdc1 --migrate



10.  Validate that all the disks have been migrated to the ASMFD.


[ root@srv10:/root: ]$ asmcmd afd_scan


[ root@srv10:/root: ]$ asmcmd afd_lsdsk

--------------------------------------------------------------------------------

Label                     Filtering   Path

================================================================================

DATA                        ENABLED   /dev/sdb1

FRA                         ENABLED   /dev/sdc1



[ root@srv10:/root: ]$ ls -ltr /dev/oracleafd/disks/

total 8

-rw-rw-r-- 1 oracle oinstall 10 Mar 25 08:33 DATA

-rw-rw-r-- 1 oracle oinstall 10 Mar 25 08:41 FRA



11.  Edit the afd_diskstring with below content. (Ensure that you put the correct afd_diskstring, otherwise ASMFD disks will not get mounted in the mount point i.e in "/dev/oracleafd/disks/" folder)

[root@hostname1 ~]# vi /etc/afd.conf

#afd_diskstring='/dev/oracleasm/disks/*'

afd_diskstring='/dev/xvd*'

afd_filtering=enable 



# --------------------------------------------------------------------------------------------

# Tips:- This value can be derived from the path value displayed while running the "asmcmd afd_lsdsk" 


12.  Repeat above steps on all the nodes of the cluster.


13.  Start the CRS on all the nodes  ( before starting the CRS, start the ACFS module)


[ root@srv10:/root: ]$ lsmod | grep acfs

[ root@srv10:/root: ]$ asmcmd -p afd_lslbl

--------------------------------------------------------------------------------

Label                     Duplicate  Path

================================================================================

DATA                                  /dev/sdb1

FRA                                   /dev/sdc1


[ root@srv10:/root: ]$ asmcmd -p afd_lsdsk

--------------------------------------------------------------------------------

Label                     Filtering   Path

================================================================================

DATA                        ENABLED   /dev/sdb1

FRA                         ENABLED   /dev/sdc1


[ root@srv10:/root: ]$ asmcmd -p afd_state

ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'srv10.dbnitro.net'


[ root@srv10:/root: ]$ lsmod | egrep "oracle|asm|afd|acfs|advm"

oracleafd             221184  0


[ root@srv10:/root: ]$ acfsload start

ACFS-9391: Checking for existing ADVM/ACFS installation.

ACFS-9129: ADVM/ACFS not installed


# Cluster

[root@hostname1 ~]# $ORACLE_HOME/bin/crsctl start crs

CRS-4123: Oracle High Availability Services has been started.


[root@hostname2 ~]# $ORACLE_HOME/bin/crsctl start crs

CRS-4123: Oracle High Availability Services has been started.


[root@hostname1 ~]# crsctl check cluster -all

**************************************************************

hostname1:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

hostname2:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************


# Standalone

[ root@srv10:/root: ]$ crsctl start has

CRS-4123: Oracle High Availability Services has been started.



14. As the Disks are migrated to ASMFD, remove ASMLIB disk path from disk discovery string. Execute the following command as a Grid Software owner from any one of the node :-


[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p dsget

parameter:/dev/oracleasm/disks/*, AFD:*

profile:/dev/oracleasm/disks/*,AFD:*


[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p dsset 'AFD:*'


[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p dsget

parameter:AFD:*

profile:AFD:*



15.  As a Grid Software owner validate that all the diskgroups are mounted properly


[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p lsdg

State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  EXTERN  N         512             512   4096  4194304     51196    48820                0           48820              0             N  DATA/

MOUNTED  EXTERN  N         512             512   4096  4194304     51196    49576                0           49576              0             N  FRA/


[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p lsdsk -G DATA

Path

AFD:DATA


[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p lsdsk -G FRA

Path

AFD:FRA



16.  Lastly validate all the disks are using ASMFD Library :- (Login into ASM instance as sysasm) 


col PATH for a20

col LIBRARY for a45

col NAME for a15

select inst_id

  , group_number grp_num

  , name

  , state

  , header_status header

  , mount_status mount

  , path

  , library

from gv$asm_disk

order by inst_id, group_number, name;


         INST_ID|         GRP_NUM|NAME           |STATE   |HEADER      |MOUNT  |PATH                |LIBRARY

----------------|----------------|---------------|--------|------------|-------|--------------------|---------------------------------------------

               1|               1|DATA_0000      |NORMAL  |MEMBER      |CACHED |AFD:DATA            |AFD Library - Generic , version 3 (KABI_V3)

               1|               2|FRA_0000       |NORMAL  |MEMBER      |CACHED |AFD:FRA             |AFD Library - Generic , version 3 (KABI_V3)



17.  Check whether the ASMFD disk is accessible even by the root user also.

[ root@srv10:/root: ]$ kfed read /dev/oracleafd/disks/DATA

KFED-00303: unable to open file '/dev/oracleafd/disks/DATA'


18. Now the migration is completed and all the disks are using ASM Filter Driver.

No comments:

Post a Comment