ACFS is neither an Oracle 12.2 new feature nor an Oracle 12.1 new feature: it is actually an Oracle 11.2 new feature.
ACFS stands for ASM Cluster File System. Oracle Restart (also knowns as single node Grid Infrastructure) is not a cluster solution, it is a software layer to manage resources like ASM disk groups, Oracle Net listeners and database instances on a single machine. How does ACFS work with Oracle Restart ? Oracle Restart does support ACFS however Oracle Restart does not take into account ACFS resources.
Note that ACFS is layered on ASM through the Oracle ADVM interface.
In this blog article I intend to show how to code Linux scripts to do what Oracle Restart does not do for ACFS.
For this blog post I have used:
$ cat /etc/os-release NAME="Oracle Linux Server" VERSION="7.3" ID="ol" VERSION_ID="7.3" PRETTY_NAME="Oracle Linux Server 7.3" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:oracle:linux:7:3:server" HOME_URL="https://linux.oracle.com/" BUG_REPORT_URL="https://bugzilla.oracle.com/" ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7" ORACLE_BUGZILLA_PRODUCT_VERSION=7.3 ORACLE_SUPPORT_PRODUCT="Oracle Linux" ORACLE_SUPPORT_PRODUCT_VERSION=7.3 $ $ORACLE_HOME/OPatch/opatch lsinv Oracle Interim Patch Installer version 12.2.0.1.6 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /u01/gi12201 Central Inventory : /u01/orainv from : /u01/gi12201/oraInst.loc OPatch version : 12.2.0.1.6 OUI version : 12.2.0.1.4 Log file location : /u01/gi12201/cfgtoollogs/opatch/opatch2018-01-27_17-51-43PM_1.log Lsinventory Output file location : /u01/gi12201/cfgtoollogs/opatch/lsinv/lsinventory2018-01-27_17-51-43PM.txt -------------------------------------------------------------------------------- Local Machine Information:: Hostname: ol7ttsa1 ARU platform id: 226 ARU platform description:: Linux x86-64 Installed Top-level Products (1): Oracle Grid Infrastructure 12c 12.2.0.1.0 There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. -------------------------------------------------------------------------------- OPatch succeeded. $ cat /etc/udev/rules.d/99-oracle-asmdevices.rules KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VBfa23f4a6-da11d28e", SYMLINK+="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VBcfa5c59e-15ba04c4", SYMLINK+="asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660" $ sqlplus / as sysdba SQL*Plus: Release 12.2.0.1.0 Production on Sat Jan 27 17:53:46 2018 Copyright (c) 1982, 2016, Oracle. All rights reserved. Connected to: Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production SYS@+ASM>column path format a20 SYS@+ASM>select name, path from v$asm_disk; NAME PATH ------------------------------ -------------------- DATA_0000 /dev/asm-disk1 RECO_0000 /dev/asm-disk2 SYS@+ASM>
Note that neither ASMLib nor ASM FD are used to manage ASM disks.
Commands with # prompt have been with run with Linux root account and commands with $ prompt have been run with Linux oracle account.
On this machine I have 2 ASM disk groups and 1 database:
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE ol7ttsa1 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE ol7ttsa1 STABLE
ora.RECO.dg
ONLINE ONLINE ol7ttsa1 STABLE
ora.asm
ONLINE ONLINE ol7ttsa1 Started,STABLE
ora.ons
OFFLINE OFFLINE ol7ttsa1 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE ol7ttsa1 STABLE
ora.db1.db
1 ONLINE ONLINE ol7ttsa1 Open,HOME=/u01/db122
01,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE ol7ttsa1 STABLE
--------------------------------------------------------------------------------
First I have run acfsload to load into Linux kernel Oracle modules needed by ACFS:
# . oraenv ORACLE_SID = [root] ? +ASM The Oracle base has been set to /u01/base [root@ol7ttsa1 ~]# which acfsload /u01/gi12201/bin/acfsload [root@ol7ttsa1 ~]# acfsload start ACFS-9391: Checking for existing ADVM/ACFS installation. ACFS-9392: Validating ADVM/ACFS installation files for operating system. ACFS-9393: Verifying ASM Administrator setup. ACFS-9308: Loading installed ADVM/ACFS drivers. ACFS-9154: Loading 'oracleoks.ko' driver. ACFS-9154: Loading 'oracleadvm.ko' driver. ACFS-9154: Loading 'oracleacfs.ko' driver. ACFS-9327: Verifying ADVM/ACFS devices. ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'. ACFS-9156: Detecting control device '/dev/ofsctl'. ACFS-9322: completed [root@ol7ttsa1 ~]# lsmod | grep pra [root@ol7ttsa1 ~]# lsmod | grep ora oracleacfs 4616192 0 oracleadvm 782336 0 oracleoks 655360 2 oracleacfs,oracleadvm
I have created an ASM volume:
$ . oraenv ORACLE_SID = [oracle] ? +ASM The Oracle base has been set to /u01/base $ asmcmd volcreate -G data -s 10G volume1 $ asmcmd volinfo -G data volume1 Diskgroup Name: DATA Volume Name: VOLUME1 Volume Device: /dev/asm/volume1-405 State: ENABLED Size (MB): 10240 Resize Unit (MB): 64 Redundancy: UNPROT Stripe Columns: 8 Stripe Width (K): 1024 Usage: Mountpath:
I have created an ACFS file system:
$ /sbin/mkfs -t acfs /dev/asm/volume1-405 mkfs.acfs: version = 12.2.0.1.0 mkfs.acfs: on-disk version = 46.0 mkfs.acfs: volume = /dev/asm/volume1-405 mkfs.acfs: volume size = 10737418240 ( 10.00 GB ) mkfs.acfs: Format complete.
I have mounted this volume as an ACFS file system:
# mkdir -p /acfsmounts/acfs1 [root@ol7ttsa1 ~]# chown oracle:dba /acfsmounts/acfs1 [root@ol7ttsa1 ~]# mount -t acfs /dev/asm/volume1-405 /acfsmounts/acfs1 [root@ol7ttsa1 ~]# df -h /acfsmounts/acfs1 Filesystem Size Used Avail Use% Mounted on /dev/asm/volume1-405 10G 265M 9.8G 3% /acfsmounts/acfs1
I have checked lsblk output:
$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 40G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 39G 0 part ├─ol-root 251:0 0 35.1G 0 lvm / └─ol-swap 251:1 0 3.9G 0 lvm [SWAP] sdb 8:16 0 40G 0 disk └─sdb1 8:17 0 40G 0 part sdc 8:32 0 10G 0 disk └─sdc1 8:33 0 10G 0 part sr0 11:0 1 1024M 0 rom asm!volume1-405 250:207361 0 10G 0 disk /acfsmounts/acfs1
I have rebooted the machine:
# systemctl reboot Connection to ol7ttsa1 closed by remote host. Connection to ol7ttsa1 closed.
I have checked that Oracle Restart looks OK:
-------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE ol7ttsa1 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE ol7ttsa1 STABLE
ora.RECO.dg
ONLINE ONLINE ol7ttsa1 STABLE
ora.asm
ONLINE ONLINE ol7ttsa1 Started,STABLE
ora.ons
OFFLINE OFFLINE ol7ttsa1 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE ol7ttsa1 STABLE
ora.db1.db
1 ONLINE ONLINE ol7ttsa1 Open,HOME=/u01/db122
01,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE ol7ttsa1 STABLE
--------------------------------------------------------------------------------
I have checked that ACFS file system is missing:
$ df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1871592 0 1871592 0% /dev tmpfs 1891012 659456 1231556 35% /dev/shm tmpfs 1891012 8568 1882444 1% /run tmpfs 1891012 0 1891012 0% /sys/fs/cgroup /dev/mapper/ol-root 36805060 28218948 8586112 77% / /dev/sda1 1038336 176476 861860 17% /boot tmpfs 378204 0 378204 0% /run/user/54321 tmpfs 378204 0 378204 0% /run/user/0 $
I have checked that no ACFS Linux modules have been loaded:
$ lsmod | grep ora $
I have tried to mount the ACFS file system:
# mount -t acfs /dev/asm/volume1-405 /acfsmounts/acfs1 mount.acfs: CLSU-00100: operating system function: open64 failed with error data: 2 mount.acfs: CLSU-00101: operating system error message: No such file or directory mount.acfs: CLSU-00103: error location: OOF_1 mount.acfs: CLSU-00104: additional error information: open64 (/dev/ofsctl) mount.acfs: ACFS-00502: Failed to communicate with the ACFS driver. Verify the ACFS driver has been loaded.
To fix this I have first reloaded ACFS drivers:
# . oraenv ORACLE_SID = [root] ? +ASM The Oracle base has been set to /u01/base # acfsload start ACFS-9391: Checking for existing ADVM/ACFS installation. ACFS-9392: Validating ADVM/ACFS installation files for operating system. ACFS-9393: Verifying ASM Administrator setup. ACFS-9308: Loading installed ADVM/ACFS drivers. ACFS-9154: Loading 'oracleoks.ko' driver. ACFS-9154: Loading 'oracleadvm.ko' driver. ACFS-9154: Loading 'oracleacfs.ko' driver. ACFS-9327: Verifying ADVM/ACFS devices. ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'. ACFS-9156: Detecting control device '/dev/ofsctl'. ACFS-9322: completed
I have checked the status of the ASM volume:
$ asmcmd volinfo -G data volume1 Diskgroup Name: DATA Volume Name: VOLUME1 Volume Device: UNKNOWN State: DISABLED Size (MB): 10240 Resize Unit (MB): 64 Redundancy: UNPROT Stripe Columns: 8 Stripe Width (K): 1024 Usage: ACFS Mountpath: /acfsmounts/acfs1
The ASM volume does exist but is disabled.
I have tried to enable the ASM volume:
$ asmcmd volenable -G data volume1 $ asmcmd volinfo -G data volume1 Diskgroup Name: DATA Volume Name: VOLUME1 Volume Device: /dev/asm/volume1-405 State: ENABLED Size (MB): 10240 Resize Unit (MB): 64 Redundancy: UNPROT Stripe Columns: 8 Stripe Width (K): 1024 Usage: ACFS Mountpath: /acfsmounts/acfs1
I have noted that enabling the volume triggers the device file creation:
$ ls -al /dev/asm/volume1-405 brwxrwx---. 1 root dba 250, 207361 Jan 27 18:13 /dev/asm/volume1-405
However at this point it is not possible to mount related ACFS file system (and I don't know why):
# mount -t acfs /dev/asm/volume1-405 /acfsmounts/acfs1/ mount.acfs: CLSU-00100: operating system function: open64 failed with error data: 1 mount.acfs: CLSU-00101: operating system error message: Operation not permitted mount.acfs: CLSU-00103: error location: OOF_1 mount.acfs: CLSU-00104: additional error information: open64 (/dev/asm/volume1-405) mount.acfs: ACFS-02017: Failed to open volume /dev/asm/volume1-405. Verify the volume exists.
Here is the systemctl script to start Oracle Restart:
# cat oracle-ohasd.service # Copyright (c) 2016, Oracle and/or its affiliates. All rights reserved. # # Oracle OHASD startup [Unit] Description=Oracle High Availability Services After=syslog.target network-online.target remote-fs.target [Service] ExecStart=/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null ExecStop=/etc/init.d/init.ohasd stop >/dev/null 2>&1 </dev/null TimeoutStopSec=60min Type=simple Restart=always # Do not kill any processes except init.ohasd after ExecStop, unless the # stop command times out. KillMode=process SendSIGKILL=yes [Install] WantedBy=multi-user.target graphical.target
In /etc/systemd/system I have created a script named acfsload.service:
[Unit] After=syslog.target Before=oracle-ohasd.service [Service] ExecStart=/u01/gi12201/bin/acfsload start -s [Install] WantedBy=default.target
I have enabled it with:
# systemctl daemon-reload # systemctl enable acfsload.service Created symlink from /etc/systemd/system/default.target.wants/acfsload.service to /etc/systemd/system/acfsload.service.
In /etc/systemd/system I have created a script named oracle-acfsvolenable.service:
[Unit] Requires=oracle-ohasd.service After=oracle-ohasd.service [Service] ExecStart=/home/oracle/scripts/acfsvolenable.sh [Install] WantedBy=default.target
I have enabled it:
# systemctl daemon-reload # systemctl enable oracle-acfsvolenable.service Created symlink from /etc/systemd/system/default.target.wants/oracle-oracle-acfsvolenable.service to /etc/systemd/system/oracle-acfsvolenable.service.
I have coded acfsvolenable.sh:
#!/bin/sh # # acfsvolenable.sh # # ----------------------------------------------------------- # wait for ASM instance while [ 1 ] do ASM_OK=$(ps -eaf | grep asm_pmon | grep -v grep | wc -l) if [ $ASM_OK -eq 1 ] then break fi sleep 1 done echo "ASM instance ready." # su - oracle -c /home/oracle/scripts/acfsvolenable_asgiowner.sh
I have given executable privilege:
# chmod u+x /home/oracle/scripts/acfsvolenable.sh
I have also coded acfsvolenable_asgiowner.sh: it must be a separate script because it must not be run by root but as Grid Infrastructure software owner (in my case this is oracle Linux account):
#!/bin/sh # # acfsvolenable_asgiowner.sh # # must be run with Grid Infrastructure owner # DGN=DATA VN=volume1 export ORAENV_ASK=NO export ORACLE_SID=+ASM . oraenv asmcmd volenable -G $DGN $VN RC=$? if [ $? -eq 0 ] then echo "In diskgroup $DGN volume $VN enabled." else echo "failed to enable in diskgroup $DGN volume $VN." fi exit $RC
I have changed ownership and privileges for this script:
# chown oracle:dba /home/oracle/scripts/acfsvolenable_asgiowner.sh # chmod u+x /home/oracle/scripts/acfsvolenable_asgiowner.sh
I have coded oracle-acfsmount.service:
[Unit] Requires=oracle-acfsvolenable.service After=oracle-acfsvolenable.service [Service] ExecStart=/home/oracle/scripts/acfsmount.sh [Install] WantedBy=default.target
I have run:
# systemctl daemon-reload # systemctl enable oracle-acfsmount.service Created symlink from /etc/systemd/system/default.target.wants/oracle-acfsmount.service to /etc/systemd/system/oracle-acfsmount.service.
I have coded acfsmount.sh:
#!/bin/sh # # acfsmount.sh # # must be run with root account # # --------------------------------------------------- # wait for acfsvolenable.sh # BD=/dev/asm/volume1-405 while [ 1 ] do if [ -b $BD ] then break fi sleep 1 done echo "$BD ready." mount -t acfs $BD /acfsmounts/acfs1
I have given executable privilege for this script:
chmod u+x /home/oracle/scripts/acfsmount.sh
To test my scripts I have rebooted my Linux machine with:
# systemctl reboot
I have checked related syslog messages:
Jan 27 19:02:27 ol7ttsa1 logger: exec /u01/gi12201/perl/bin/perl -I/u01/gi12201/perl/lib /u01/gi12201/bin/crswrapexece.pl /u01/gi12201/crs/install/s_crsconfig_ol7ttsa1_env.txt /u01/gi12201/bin/ohasd.bin "reboot" Jan 27 19:02:27 ol7ttsa1 systemd: Started Postfix Mail Transport Agent. Jan 27 19:02:27 ol7ttsa1 systemd: Reached target Multi-User System. Jan 27 19:02:27 ol7ttsa1 systemd: Starting Multi-User System. Jan 27 19:02:27 ol7ttsa1 systemd: Starting Update UTMP about System Runlevel Changes... Jan 27 19:02:27 ol7ttsa1 systemd: Started Update UTMP about System Runlevel Changes. Jan 27 19:02:30 ol7ttsa1 kdumpctl: kexec: loaded kdump kernel Jan 27 19:02:30 ol7ttsa1 kdumpctl: Starting kdump: [OK] Jan 27 19:02:30 ol7ttsa1 systemd: Started Crash recovery kernel arming. Jan 27 19:02:30 ol7ttsa1 systemd: Startup finished in 402ms (kernel) + 2.864s (initrd) + 12.067s (userspace) = 15.335s. Jan 27 19:02:37 ol7ttsa1 su: (to oracle) root on none Jan 27 19:02:37 ol7ttsa1 systemd: Started Session c3 of user oracle. Jan 27 19:02:37 ol7ttsa1 systemd: Starting Session c3 of user oracle. Jan 27 19:02:37 ol7ttsa1 journal: Oracle Clusterware: 2018-01-27 19:02:37.823#012[(1340)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 1340 Jan 27 19:02:37 ol7ttsa1 clsecho: /etc/init.d/init.ohasd: Writing 1340 to oracle-ohasd tasks Jan 27 19:02:37 ol7ttsa1 init.ohasd: /etc/init.d/init.ohasd: Writing 1340 to oracle-ohasd tasks Jan 27 19:02:58 ol7ttsa1 acfsvolenable.sh: ASM instance ready. Jan 27 19:02:58 ol7ttsa1 su: (to oracle) root on none Jan 27 19:02:58 ol7ttsa1 systemd: Started Session c4 of user oracle. Jan 27 19:02:58 ol7ttsa1 systemd: Starting Session c4 of user oracle. Jan 27 19:02:58 ol7ttsa1 acfsvolenable.sh: The Oracle base has been set to /u01/base Jan 27 19:02:59 ol7ttsa1 kernel: asm_rbal_+asm: sending ioctl 2285 to a partition! Jan 27 19:02:59 ol7ttsa1 kernel: asm_rbal_+asm: sending ioctl 2285 to a partition! Jan 27 19:02:59 ol7ttsa1 kernel: asm_rbal_+asm: sending ioctl 2285 to a partition! Jan 27 19:02:59 ol7ttsa1 kernel: asm_rbal_+asm: sending ioctl 2285 to a partition! Jan 27 19:03:05 ol7ttsa1 acfsmount.sh: /dev/asm/volume1-405 ready. Jan 27 19:03:06 ol7ttsa1 acfsvolenable.sh: In diskgroup DATA volume volume1 enabled. Jan 27 19:03:11 ol7ttsa1 kernel: ora_rbal_db1: sending ioctl 2285 to a partition! Jan 27 19:03:11 ol7ttsa1 kernel: ora_rbal_db1: sending ioctl 2285 to a partition! Jan 27 19:03:52 ol7ttsa1 systemd: Created slice user-0.slice. Jan 27 19:03:52 ol7ttsa1 systemd: Starting user-0.slice. Jan 27 19:03:52 ol7ttsa1 systemd: Started Session 1 of user root. Jan 27 19:03:52 ol7ttsa1 systemd-logind: New session 1 of user root. Jan 27 19:03:52 ol7ttsa1 systemd: Starting Session 1 of user root.
Even if acfsmount.sh seems to be completed before acfsvolenable.sh, I don't think this is the case.
I have checked that df shows that the ACFS file system is now automatically mounted:
Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1871592 0 1871592 0% /dev tmpfs 1891012 659456 1231556 35% /dev/shm tmpfs 1891012 8780 1882232 1% /run tmpfs 1891012 0 1891012 0% /sys/fs/cgroup /dev/mapper/ol-root 36805060 28545272 8259788 78% / /dev/sda1 1038336 176476 861860 17% /boot tmpfs 378204 0 378204 0% /run/user/54321 /dev/asm/volume1-405 10485760 271356 10214404 3% /acfsmounts/acfs1 tmpfs 378204 0 378204 0% /run/user/0
This shows that it is possible to use Linux shell scripts to manage ACFS resources for Oracle Restart.