# --------------------------------------------------------------------------------------------
# ASMFD : How to Migrate ASM Diskgroups from ASMLIB to ASMFD (ASM Filter Driver) on Oracle Grid Infrastructure (RAC) (Doc ID 2172754.1)
# --------------------------------------------------------------------------------------------
# Applies to:
Oracle Database Backup Service - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later Linux x86-64
# --------------------------------------------------------------------------------------------
# What is Oracle ASM Filter Driver (Oracle ASMFD) ?
Oracle ASM Filter Driver (Oracle ASMFD) is a kernel module that resides in the I/O path of the Oracle ASM disks. Oracle ASM uses the filter driver to validate write I/O requests to Oracle ASM disks.
The Oracle ASMFD simplifies the configuration and management of disk devices by eliminating the need to rebind disk devices used with Oracle ASM each time the system is restarted.
The Oracle ASM Filter Driver rejects any I/O requests that are invalid. This action eliminates accidental overwrites of Oracle ASM disks that would cause corruption in the disks and files within the disk group. For example, the Oracle ASM Filter Driver filters out all non-Oracle I/Os which could cause accidental overwrites.
Oracle ASM Filter Driver (Oracle ASMFD) is installed with an Oracle Grid Infrastructure installation. If you have an existing Oracle ASM library driver (Oracle ASMLIB) configuration, then this document will guide you to migrate the existing ASM diskgroup disks which are stamped with ASMLib to ASMFD (ASM Filter Driver).
# --------------------------------------------------------------------------------------------
# Solution
Following assumptions are made before we proceed with the next steps
1. Please check the existing environment is certified or not. Check below document
ASMFD (ASM Filter Driver) Support on OS Platforms (Certification Matrix).(Doc ID 2034681.1)
2. It is recommended to apply latest PSU for successful migration.
3. Take the backup of the database before proceeding this activity.
# --------------------------------------------------------------------------------------------
# Step by Step Procedure
1. As a root user run below command to generate the which ASM Disk is associated with the OS disk. In a multipath configuration, please use the multipath name
[ root@srv10:/home: ]$ oracleasm listdisks | xargs oracleasm querydisk -p
Disk "DATA" is a valid ASM disk /dev/sdb1: LABEL="DATA" TYPE="oracleasm"
Disk "FRA" is a valid ASM disk /dev/sdc1: LABEL="FRA" TYPE="oracleasm"
2. As a Grid Infrastructure (GI) software owner, get the present ASM diskstring value
[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p dsget
parameter:/dev/oracleasm/disks/*
profile:/dev/oracleasm/disks/*
[ +ASM ]|[ oracle@srv10:/home/oracle: ]$
In case the ASMLIB disk discovery string is NOT /dev/oracleasm/disks/* but "ORCL:*".
grid@hostname1 ~]$ asmcmd dsget
parameter:ORCL:*
profile:ORCL:*
3. As a GI owner update the Oracle ASM disk discovery string to enable Oracle ASMFD to discover devices in the future.
[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p dsset '/dev/oracleasm/disks/*','AFD:*'
[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p dsget
parameter:/dev/oracleasm/disks/*, AFD:*
profile:/dev/oracleasm/disks/*,AFD:*
In case the ASMLIB disk discovery string is NOT /dev/oracleasm/disks/* but "ORCL:*".
[grid@hostname1 ~]$ asmcmd dsset 'ORCL:*','AFD:*'
[grid@hostname1 ~]$ asmcmd dsget
parameter:ORCL:*, AFD:*
profile:ORCL:*,AFD:*
4. As a root user stop the cluster accessing the shared ASMLib disks
# Cluster
[ root@srv10:/root: ]$ crsctl stop crs
# Standalone
[ root@srv10:/root: ]$ crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'srv10'
CRS-2673: Attempting to stop 'ora.dbprod.db' on 'srv10'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'srv10'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'srv10' succeeded
CRS-2677: Stop of 'ora.dbprod.db' on 'srv10' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'srv10'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'srv10'
CRS-2677: Stop of 'ora.DATA.dg' on 'srv10' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'srv10' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'srv10'
CRS-2677: Stop of 'ora.asm' on 'srv10' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'srv10'
CRS-2677: Stop of 'ora.evmd' on 'srv10' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'srv10'
CRS-2677: Stop of 'ora.cssd' on 'srv10' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'srv10' has completed
CRS-4133: Oracle High Availability Services has been stopped.
5. Check and Stop the oracleasm driver and acfsload as root user
[ root@srv10:/root: ]$ oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[ root@srv10:/root: ]$ oracleasm exit
Unmounting ASMlib driver filesystem: /dev/oracleasm
Unloading module "oracleasm": oracleasm
[ root@srv10:/root: ]$ ls -ltr /dev/oracleasm/
total 0
[ root@srv10:/root: ]$ lsmod | grep acfs
oracleacfs 3498241 2
oracleoks 503994 2 oracleacfs,oracleadvm
[ root@srv10:/root: ]$ acfsload stop
[ root@srv10:/root: ]$ lsmod | grep acfs
# --------------------------------------------------------------------------------------------
# Note: It is recommended to remove the ASMLib related RPMs to avoid any conflict.
[ root@srv10:/root: ]$ rpm -qa | grep asm
oracleasmlib-2.0.12-1.el7.x86_64
kmod-oracleasm-2.0.8-28.0.1.el7.x86_64
oracleasm-support-2.1.11-2.el7.x86_64
[ root@srv10:/root: ]$ rpm -e oracleasm-support-2.1.11-2.el7.x86_64 kmod-oracleasm-2.0.8-28.0.1.el7.x86_64 oracleasmlib-2.0.12-1.el7.x86_64
warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave
6. As root, configure Oracle ASMFD to filter at the node level:
[ root@srv10:/root: ]$ afdroot version_check
AFD-616: Valid AFD distribution media detected at: '/u01/app/19.3.0.1/grid/usm/install/Oracle/EL7UEK/x86_64/5.4.17-2011.0.7/5.4.17-2011.0.7-x86_64/bin'
[ root@srv10:/root: ]$ afddriverstate supported
AFD-9200: Supported
[ root@srv10:/root: ]$ asmcmd afd_configure
AFD-627: AFD distribution files found.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-9294: updating file /etc/sysconfig/oracledrivers.conf
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.
7. As the Oracle Grid Infrastructure / root owner verify the status of Oracle ASMFD:
[ root@srv10:/root: ]$ asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'srv10.dbnitro.net'
# Case is disabled
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'DISABLED' on host 'hostname1' <<<< Filtering is disabled
8. As a root user set the AFD diskstring and enable the Filtering
[ root@srv10:/root: ]$ asmcmd -p afd_filter -e
[ root@srv10:/root: ]$ asmcmd -p afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'srv10.dbnitro.net' <<<< Filtering is enabled
9. As a root user, with the help of the data gathered in step 1, migrate (label) all the ASMLib disks
$ORACLE_HOME/bin/asmcmd afd_label <OLD ASMLib Disk Label Name> <Disk Path> --migrate
[ root@srv10:/root: ]$ cat /etc/oracleafd.conf
afd_diskstring='/dev/sd*'
afd_filtering=enable
[ root@srv10:/root: ]$ asmcmd afd_scan
[ root@srv10:/root: ]$ asmcmd afd_lsdsk
[ root@srv10:/root: ]$ asmcmd -p afd_label DATA /dev/sdb1 --migrate
[ root@srv10:/root: ]$ asmcmd -p afd_label FRA /dev/sdc1 --migrate
10. Validate that all the disks have been migrated to the ASMFD.
[ root@srv10:/root: ]$ asmcmd afd_scan
[ root@srv10:/root: ]$ asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label Filtering Path
================================================================================
DATA ENABLED /dev/sdb1
FRA ENABLED /dev/sdc1
[ root@srv10:/root: ]$ ls -ltr /dev/oracleafd/disks/
total 8
-rw-rw-r-- 1 oracle oinstall 10 Mar 25 08:33 DATA
-rw-rw-r-- 1 oracle oinstall 10 Mar 25 08:41 FRA
11. Edit the afd_diskstring with below content. (Ensure that you put the correct afd_diskstring, otherwise ASMFD disks will not get mounted in the mount point i.e in "/dev/oracleafd/disks/" folder)
[root@hostname1 ~]# vi /etc/afd.conf
#afd_diskstring='/dev/oracleasm/disks/*'
afd_diskstring='/dev/xvd*'
afd_filtering=enable
# --------------------------------------------------------------------------------------------
# Tips:- This value can be derived from the path value displayed while running the "asmcmd afd_lsdsk"
12. Repeat above steps on all the nodes of the cluster.
13. Start the CRS on all the nodes ( before starting the CRS, start the ACFS module)
[ root@srv10:/root: ]$ lsmod | grep acfs
[ root@srv10:/root: ]$ asmcmd -p afd_lslbl
--------------------------------------------------------------------------------
Label Duplicate Path
================================================================================
DATA /dev/sdb1
FRA /dev/sdc1
[ root@srv10:/root: ]$ asmcmd -p afd_lsdsk
--------------------------------------------------------------------------------
Label Filtering Path
================================================================================
DATA ENABLED /dev/sdb1
FRA ENABLED /dev/sdc1
[ root@srv10:/root: ]$ asmcmd -p afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'srv10.dbnitro.net'
[ root@srv10:/root: ]$ lsmod | egrep "oracle|asm|afd|acfs|advm"
oracleafd 221184 0
[ root@srv10:/root: ]$ acfsload start
ACFS-9391: Checking for existing ADVM/ACFS installation.
ACFS-9129: ADVM/ACFS not installed
# Cluster
[root@hostname1 ~]# $ORACLE_HOME/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@hostname2 ~]# $ORACLE_HOME/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@hostname1 ~]# crsctl check cluster -all
**************************************************************
hostname1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
hostname2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
# Standalone
[ root@srv10:/root: ]$ crsctl start has
CRS-4123: Oracle High Availability Services has been started.
14. As the Disks are migrated to ASMFD, remove ASMLIB disk path from disk discovery string. Execute the following command as a Grid Software owner from any one of the node :-
[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p dsget
parameter:/dev/oracleasm/disks/*, AFD:*
profile:/dev/oracleasm/disks/*,AFD:*
[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p dsset 'AFD:*'
[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p dsget
parameter:AFD:*
profile:AFD:*
15. As a Grid Software owner validate that all the diskgroups are mounted properly
[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 512 4096 4194304 51196 48820 0 48820 0 N DATA/
MOUNTED EXTERN N 512 512 4096 4194304 51196 49576 0 49576 0 N FRA/
[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p lsdsk -G DATA
Path
AFD:DATA
[ +ASM ]|[ oracle@srv10:/home/oracle: ]$ asmcmd -p lsdsk -G FRA
Path
AFD:FRA
16. Lastly validate all the disks are using ASMFD Library :- (Login into ASM instance as sysasm)
col PATH for a20
col LIBRARY for a45
col NAME for a15
select inst_id
, group_number grp_num
, name
, state
, header_status header
, mount_status mount
, path
, library
from gv$asm_disk
order by inst_id, group_number, name;
INST_ID| GRP_NUM|NAME |STATE |HEADER |MOUNT |PATH |LIBRARY
----------------|----------------|---------------|--------|------------|-------|--------------------|---------------------------------------------
1| 1|DATA_0000 |NORMAL |MEMBER |CACHED |AFD:DATA |AFD Library - Generic , version 3 (KABI_V3)
1| 2|FRA_0000 |NORMAL |MEMBER |CACHED |AFD:FRA |AFD Library - Generic , version 3 (KABI_V3)
17. Check whether the ASMFD disk is accessible even by the root user also.
[ root@srv10:/root: ]$ kfed read /dev/oracleafd/disks/DATA
KFED-00303: unable to open file '/dev/oracleafd/disks/DATA'
18. Now the migration is completed and all the disks are using ASM Filter Driver.