Partition EMC LUNs to allocate more disk space for Oracle

Hi,

As a Junior DBA, as of the moment, you may find my posts a little easy, but I would like to share these easy things to all newbies to Oracle. I have a recent activity with one of our clients. Here is the scenario and I would like to share it:

Problem:
Oracle dump files are very important to this client because they are using it for troubleshooting purposes on their RAC environment. Oracle's bdump, cdump and udump grows so fast that they need to change the disk to allocate bigger disk space. These logs make /u01 ,which is the mount for for their ORACLE_HOME, grow big and reaches 99%.

Goal:
1. To partition the presented disk done by the storage engineer
2. To allot one disk per node since the disks are visible on the three nodes
3. Move the log files to these new mount points
4. Make sure that these newly partitioned disks are accessible by Oracle.


Here's what I did:

Findings:

-       EMC was able to create three LUNs of 20 GB each.
-       The three LUNs are labeled as: /dev/emcpowerq, /dev/emcpowerr and /dev/emcpowers are visible to the three nodes but only one LUN will be used per node.
-       /u01 mount points across the nodes are always above threshold which can cause database to quit working.

Action Items:

-       Checked and verified is the presented LUNs are accessible and visible using fdisk. By issuing
fdisk –l, the output is shown below:

Disk /dev/emcpowerq: 21.4 GB, 21474836480 bytes
64 heads, 32 sectors/track, 20480 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/emcpowerq doesn't contain a valid partition table

Disk /dev/emcpowerr: 21.4 GB, 21474836480 bytes
64 heads, 32 sectors/track, 20480 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/emcpowerr doesn't contain a valid partition table

Disk /dev/emcpowers: 21.4 GB, 21474836480 bytes
64 heads, 32 sectors/track, 20480 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/emcpowers doesn't contain a valid partition table



-       Mapped the distributions of the allocated 21.4GB LUNs for the three dumpfiles: bdump, cdump and udump according to their existing utilization:
total 21,913.6 MB
bdump 10,240 MB
cdump 4,096 MB
udump 7,168 MB



-       For rac1-1:
Disk /dev/emcpowerq: 21.4 GB, 21474836480 bytes
64 heads, 32 sectors/track, 20480 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

root# fdisk /dev/emcpowerq

The sequence of the steps in partitioning the disk and formatting it is shown below:

# fdisk /dev/emcpowerq
   n --new patition
   p --partition
   1 --partition 1
   <Enter>
   +10240M --size of the partition
   n --new partition
   p --partition
   2 -- partition 2
   <Enter>
   +4096M --size of the partition
   n --new partition
   p --partition
   <Enter> --default 3rd partition
   <Enter>
   p --print the partitions made
   w --write the new partition table and save




# fdisk –l
Disk /dev/emcpowerq: 21.4 GB, 21474836480 bytes
64 heads, 32 sectors/track, 20480 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

         Device Boot      Start         End      Blocks   Id  System
/dev/emcpowerq1               1        9767    10001392   83  Linux
/dev/emcpowerq2            9768       13674     4000768   83  Linux
/dev/emcpowerq3           13675       20480     6969344   83  Linux

-       Format the partitioned LUN to ext3 file system:
[root@rac1-1 ~]# mkfs -t ext3 /dev/emcpowerq1
mke2fs 1.35 (28-Feb-2004)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1251712 inodes, 2500348 blocks
125017 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2562719744
77 block groups
32768 blocks per group, 32768 fragments per group
16256 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@rac1-1 ~]# mkfs -t ext3 /dev/emcpowerq2
mke2fs 1.35 (28-Feb-2004)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
500960 inodes, 1000192 blocks
50009 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1027604480
31 block groups
32768 blocks per group, 32768 fragments per group
16160 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 22 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@rac1-1 ~]# mkfs -t ext3 /dev/emcpowerq3
mke2fs 1.35 (28-Feb-2004)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
872640 inodes, 1742336 blocks
87116 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1786773504
54 block groups
32768 blocks per group, 32768 fragments per group
16160 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

-       Created temporary mount points to test the mounting and unmounting of the newly partitioned devices:

# /mnt] mkdir bdump cdump udump
# mount /dev/emcpowerq1 /mnt/bdump
# mount /dev/emcpowerq2 /mnt/cdump
# mount /dev/emcpowerq3 /mnt/udump

# df –h
Filesystem            Size  Used Avail Use% Mounted on
/dev/emcpowerq1       9.4G  841M  8.1G  10% /mnt/bdump
/dev/emcpowerq2       3.8G   40M  3.6G   2% /mnt/bdump
/dev/emcpowerq3       6.6G  401M  5.9G   7% /mnt/bdump

# umount /dev/emcpowerq1
# umount /dev/emcpowerq2
# umount /dev/emcpowerq3

# df –h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda7            1012M  309M  652M  33% /
/dev/sda1             114M   14M   95M  13% /boot
none                  2.0G     0  2.0G   0% /dev/shm
/dev/mapper/vgsystem-lvhome
                      8.9G  3.8G  4.7G  45% /home
/dev/sda5             2.0G  264K  2.0G   1% /mnt/sda7
/dev/mapper/vgsystem-lvtmp
                     1008M  110M  847M  12% /tmp
/dev/mapper/vgsystem-lvusr
                      4.4G  2.5G  1.8G  59% /usr
/dev/mapper/vgsystem-lvvar
                      2.0G  892M 1023M  47% /var
/dev/mapper/vgsystem-lvu01
                       11G  8.7G  1.7G  84% /u01
/dev/emcpowerl1       500M  120M  380M  24% /u02
/dev/emcpowerg1       401G  296G  106G  74% /u03

-       All Oracle-related services were shutdown. Verified that no CRS binaries are up and running.
-       Move the dump files located in $ORACLE_BASE/admin/prod to the corresponding location in the /mnt directory. After copying, renamed the dump files to bdump-bak, cdump-bak and udump-bak and create new bdump, cdump and udump on the same directory.
-       Mounted the emc LUNs on these permanent directories and tested its mounting and unmounting capabilities. Changed the ownership of these permanent mount points using:
chown –R oracle:oinstall
-       Enables the CRS using crsctl enable crs then rebooted the machine.
-       After ensuring that the mountpoints are intact and the LUNs ID’s are permanent, the fstab was updated to load the newly partitioned volumes upon system boot up:

# vi /etc/fstab
[root@rac1-1 ~]# cat /etc/fstab
# This file is edited by fstab-sync - see 'man fstab-sync' for details
LABEL=/                 /                       ext3    defaults        1 1
LABEL=/boot1            /boot                   ext3    defaults        1 2
none                    /dev/pts                devpts  gid=5,mode=620  0 0
none                    /dev/shm                tmpfs   defaults        0 0
/dev/vgsystem/lvhome    /home                   ext3    defaults        1 2
/dev/sda5               /mnt/sda7               vfat    defaults        0 0
none                    /proc                   proc    defaults        0 0
none                    /sys                    sysfs   defaults        0 0
/dev/vgsystem/lvtmp     /tmp                    ext3    defaults        1 2
/dev/vgsystem/lvusr     /usr                    ext3    defaults        1 2
/dev/vgsystem/lvvar     /var                    ext3    defaults        1 2
LABEL=SWAP-sda6         swap                    swap    defaults        0 0
LABEL=SWAP-sda2         swap                    swap    defaults        0 0
LABEL=SWAP-sda3         swap                    swap    defaults        0 0



# Oracle Cluster File System 2 (OCFS) file systems
/dev/emcpowerl1 /u02 ocfs2 _netdev,datavolume,nointr 0 0
/dev/emcpowerg1 /u03 ocfs2 _netdev 0 0

/dev/vgsystem/lvu01 /u01 ext3 defaults 1 2


# Oracle dump file system (bdump, cdump, udump)
/dev/emcpowerq1         /u01/app/oracle/admin/prod/bdump        ext3    defaults        1 2
/dev/emcpowerq2         /u01/app/oracle/admin/prod/cdump        ext3    defaults        1 2
/dev/emcpowerq3         /u01/app/oracle/admin/prod/udump        ext3    defaults        1 2

/dev/hda                /media/cdrom            auto    pamconsole,fscontext=system_u:object_r:removable_t,exec,noauto,managed 0 0

-       The machine was rebooted and restarted all Oracle related services. Verified that the startup and shutdown activity of the database are recorded in the alert logs. Created and dropped a test tablespace to make sure that it will be recorded in the logs are usual.
-       Housekeeping.

-       For rac1-2 and rac1-3:

The same process was done and the outputs are the following:

RAC1-2
Disk /dev/emcpowerr: 21.4 GB, 21474836480 bytes
64 heads, 32 sectors/track, 20480 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/emcpowerr doesn't contain a valid partition table

      Disk /dev/emcpowerr: 21.4 GB, 21474836480 bytes
64 heads, 32 sectors/track, 20480 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

         Device Boot      Start         End      Blocks   Id  System
/dev/emcpowerr1               1        9767    10001392   83  Linux
/dev/emcpowerr2            9768       13674     4000768   83  Linux
/dev/emcpowerr3           13675       20480     6969344   83  Linux


RAC1-3
Disk /dev/emcpowers: 21.4 GB, 21474836480 bytes
64 heads, 32 sectors/track, 20480 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/emcpowers doesn't contain a valid partition table




Disk /dev/emcpowers: 21.4 GB, 21474836480 bytes
64 heads, 32 sectors/track, 20480 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

         Device Boot      Start         End      Blocks   Id  System
/dev/emcpowers1               1        9767    10001392   83  Linux
/dev/emcpowers2            9768       13674     4000768   83  Linux
/dev/emcpowers3           13675       20480     6969344   83  Linux

The fstab for the two nodes are shown below:

[root@rac1-2 ~]# cat /etc/fstab
# This file is edited by fstab-sync - see 'man fstab-sync' for details
LABEL=/                 /                       ext3    defaults        1 1
LABEL=/boot1            /boot                   ext3    defaults        1 2
none                    /dev/pts                devpts  gid=5,mode=620  0 0
none                    /dev/shm                tmpfs   defaults        0 0
/dev/vgsystem/lvhome    /home                   ext3    defaults        1 2
/dev/sda7         /mnt/sda7               vfat    defaults        0 0
none                    /proc                   proc    defaults        0 0
none                    /sys                    sysfs   defaults        0 0
/dev/vgsystem/lvtmp     /tmp                    ext3    defaults        1 2
/dev/vgsystem/lvusr     /usr                    ext3    defaults        1 2
/dev/vgsystem/lvvar     /var                    ext3    defaults        1 2
LABEL=SWAP-sda5         swap                    swap    defaults        0 0
LABEL=SWAP-sda2         swap                    swap    defaults        0 0
LABEL=SWAP-sda3         swap                    swap    defaults        0 0

# Oracle Cluster File System 2 (OCFS) file systems
/dev/emcpowerl1 /u02 ocfs2 _netdev,datavolume,nointr 0 0
/dev/emcpowerg1 /u03 ocfs2 _netdev 0 0


/dev/vgsystem/lvu01 /u01 ext3 defaults 1 2

/dev/vgsystem/lvbackups /var/backups ext3 defaults 1 2

# Oracle dump file system (bdump,cdump and udump)
/dev/emcpowerr1       /u01/app/oracle/admin/prod/bdump  ext3    defaults        1 2
/dev/emcpowerr2       /u01/app/oracle/admin/prod/cdump  ext3    defaults        1 2
/dev/emcpowerr3       /u01/app/oracle/admin/prod/udump  ext3    defaults        1 2

/dev/hda                /media/cdrom            auto    pamconsole,fscontext=system_u:object_r:removable_t,exec,noauto,managed 0 0






[root@rac1-3 ~]# cat /etc/fstab
# This file is edited by fstab-sync - see 'man fstab-sync' for details
LABEL=/                 /                       ext3    defaults        1 1
LABEL=/boot1            /boot                   ext3    defaults        1 2
none                    /dev/pts                devpts  gid=5,mode=620  0 0
none                    /dev/shm                tmpfs   defaults        0 0
/dev/vgsystem/lvhome    /home                   ext3    defaults        1 2
/dev/sda6               /mnt/sda6               vfat    defaults        0 0
/dev/vgsystem/lvopt     /opt                    ext3    defaults        1 2
none                    /proc                   proc    defaults        0 0
none                    /sys                    sysfs   defaults        0 0
/dev/vgsystem/lvtmp     /tmp                    ext3    defaults        1 2
/dev/vgsystem/lvu01     /u01                    ext3    defaults        1 2
/dev/vgsystem/lvusr     /usr                    ext3    defaults        1 2
/dev/vgsystem/lvvar     /var                    ext3    defaults        1 2
/dev/vgsystem/lvbackups /var/backups            ext3    defaults        1 2
LABEL=SWAP-sda3         swap                    swap    defaults        0 0
LABEL=SWAP-sda2         swap                    swap    defaults        0 0

# Oracle Cluster File System 2 (OCFS2) file systems
/dev/emcpowerl1 /u02 ocfs2 _netdev,datavolume,nointr 0 0
/dev/emcpowerg1 /u03 ocfs2 _netdev 0 0

# Additional swap
/dev/vgsystem/lvswap swap swap defaults 0 0

#Oracle dumo file system (bdump, cdump, udump)
/dev/emcpowers1         /u01/app/oracle/admin/prod/bdump        ext3    defaults        1 2
/dev/emcpowers2         /u01/app/oracle/admin/prod/cdump        ext3    defaults        1 2
/dev/emcpowers3         /u01/app/oracle/admin/prod/udump        ext3    defaults        1 2

/dev/hda                /media/cdrom            auto    pamconsole,fscontext=system_u:object_r:removable_t,exec,noauto,managed 0 0


I hope you find this a good stuff to read.




Gilbert S Dela Cruz
Oracle Database Administrator
Oracle Database 11g Certified Associate
Registered Electronics Engineer
Member IECEP Philippines Inc.



Comments

Post a Comment

Popular posts from this blog

RMAN Restoration to New Server with Different Directory and New Database Name

In this case, I have an existing production database with RMAN Level 0 backup: CONFIGURE CONTROLFILE AUTOBACKUP ON; CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO 'C:\flash_recovery_area\galaxy\backupsets\autobackup\SAMSUNG-ctl_%F'; CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS; CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT 'C:\flash_recovery_area\galaxy\backupsets\level0\level0-%d-%T-%U' MAXPIECESIZE 4G; DELETE NOPROMPT ARCHIVELOG ALL COMPLETED BEFORE 'SYSDATE - 7'; SHOW ALL; REPORT SCHEMA; LIST BACKUP OF DATABASE; REPORT NEED BACKUP; REPORT UNRECOVERABLE; LIST EXPIRED BACKUP BY FILE; LIST ARCHIVELOG ALL; REPORT OBSOLETE; CROSSCHECK BACKUP DEVICE TYPE DISK; CROSSCHECK COPY OF ARCHIVELOG ALL; DELETE NOPROMPT EXPIRED BACKUP DEVICE TYPE DISK; DELETE NOPROMPT OBSOLETE DEVICE TYPE DISK; RUN { sql 'alter system archive log current'; BACKUP INCREMENTAL LEVEL 0 DATABASE PLUS ARCHIVELOG DELETE INPUT; } LIST BACKUP ...
Read more

[Script] Tablespace Usage Alert

Hi, This is another variation of the previous scripts presented in this blog site. The building blocks of this script is the same as the Disk Space and ASM Diskgroup Space Monitoring scripts. This will alert you via email if a certain tablespace  reaches the set threshold. Let's share and make others love Oracle, too. For tablespace monitoring alert scripts. We have the following: Tablespace Monitoring Script You will notice that there is a pattern in the scripts creation presented in this blog.  Part 1: Report Generation : Here it is the tablespace report Part 2: Report output string process : Scripts are created to process further report into columns. Conditional statement was created to test the values in the report. Part 3: Check if the file generates alert : Email is sent if entities met the threshold. # # # # # #----- Start of Script -----###### #!/bin/bash  # Report tablespace usage  sqlplus "/ as sysdba" << EOF SET LINES 700...
Read more

[Script] ASM Diskgroup Space Usage Alert

Hi Again, Another important task that a DBA should do is to monitor the ASM diskgroup usage. We do not just add space to tablespaces, create more tablespaces or resize datafiles to increase storage without checking if there is space to our ASM diskgroup. Also, when we perform RMAN backups stored in ASM or Datapump Exports stored in ASM, we have to monitor and get alerts if we are running out of space. This is a simple monitoring script. I hope this could help you. Let's share and make others love Oracle, too. ASM Diskgroup  Space Monitoring ######--- Start of Script -----###### #!/bin/bash  # Report ASM Diskgroup space usage  #NOTES:  # 1. There is really a slash on the v$asm_diskgroup so Unix interpreter will treat dollar sign as a literal $ # 2. Set your tablespace threshold by changing the value of the AND clause  " >= 80" sqlplus "/ as sysdba" << EOF SET LINES 700 SET FEEDBACK OFF SET TRIMSPOOL ON SET ECHO OFF SPOOL /tmp/ASMs...
Read more