--- Get a list of block devices from kernel message ------------ www.6park.com
[root@vmvc01 ~]# dmesg | grep MB
PCI quirk: region 1040-104f claimed by PIIX4 SMB
 target0:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
SCSI device sda: 125829120 512-byte hdwr sectors (64425 MB)
SCSI device sda: 125829120 512-byte hdwr sectors (64425 MB)
 target0:0:1: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
SCSI device sdb: 62914560 512-byte hdwr sectors (32212 MB)
SCSI device sdb: 62914560 512-byte hdwr sectors (32212 MB)
 target0:0:2: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
SCSI device sdc: 251658240 512-byte hdwr sectors (128849 MB)
SCSI device sdc: 251658240 512-byte hdwr sectors (128849 MB)
piix4_smbus 0000:00:07.3: Host SMBus controller not enabled!
SCSI device sdb: 62914560 512-byte hdwr sectors (32212 MB)
SCSI device sdb: 62914560 512-byte hdwr sectors (32212 MB) www.6park.com
--- Partition my drive into 5 parts --------------------------- www.6park.com

[root@vmvc01 ~]# fdisk /dev/sdc www.6park.com
The number of cylinders for this disk is set to 15665.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK) www.6park.com
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-15665, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-15665, default 15665): +24576M www.6park.com
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (2990-15665, default 2990):
Using default value 2990
Last cylinder or +size or +sizeM or +sizeK (2990-15665, default 15665): +24576M www.6park.com
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (5979-15665, default 5979):
Using default value 5979
Last cylinder or +size or +sizeM or +sizeK (5979-15665, default 15665): +24576M www.6park.com
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
e
Selected partition 4
First cylinder (8968-15665, default 8968):
Using default value 8968
Last cylinder or +size or +sizeM or +sizeK (8968-15665, default 15665):
Using default value 15665 www.6park.com
Command (m for help): n
First cylinder (8968-15665, default 8968):
Using default value 8968
Last cylinder or +size or +sizeM or +sizeK (8968-15665, default 15665): +24576M www.6park.com
Command (m for help): n
First cylinder (11957-15665, default 11957):
Using default value 11957
Last cylinder or +size or +sizeM or +sizeK (11957-15665, default 15665): +24576M www.6park.com
Command (m for help): p www.6park.com
Disk /dev/sdc: 128.8 GB, 128849018880 bytes
255 heads, 63 sectors/track, 15665 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes www.6park.com
   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1        2989    24009111   83  Linux
/dev/sdc2            2990        5978    24009142+  83  Linux
/dev/sdc3            5979        8967    24009142+  83  Linux
/dev/sdc4            8968       15665    53801685    5  Extended
/dev/sdc5            8968       11956    24009111   83  Linux
/dev/sdc6           11957       14945    24009111   83  Linux www.6park.com
Command (m for help): w
The partition table has been altered! www.6park.com
Calling ioctl() to re-read partition table.
Syncing disks. www.6park.com
--- Create a raid5 device ----------------------------------------------- www.6park.com
[root@vmvc01 ~]# mdadm --create /dev/md0 --chunk=64 --level=raid5 --raid-devices=5 /dev/sdc1 /dev/sdc2 /dev/sdc3 /dev/sdc5 /dev/sdc6
mdadm: array /dev/md0 started. www.6park.com
[root@vmvc01 ~]# more /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdc6[5] sdc5[3] sdc3[2] sdc2[1] sdc1[0]
      96036096 blocks level 5, 64k chunk, algorithm 2 [5/4] [UUUU_]
      [>....................]  recovery =  0.1% (26112/24009024) finish=396.6min speed=1004K/sec www.6park.com
unused devices:  www.6park.com
[root@vmvc01 ~]# mdadm --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Sun May 29 15:15:54 2011
     Raid Level : raid5
     Array Size : 96036096 (91.59 GiB 98.34 GB)
  Used Dev Size : 24009024 (22.90 GiB 24.59 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 0
    Persistence : Superblock is persistent www.6park.com
    Update Time : Sun May 29 15:15:54 2011
          State : clean, degraded, recovering
 Active Devices : 4
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 1 www.6park.com
         Layout : left-symmetric
     Chunk Size : 64K www.6park.com
 Rebuild Status : 0% complete www.6park.com
           UUID : cbe75ee2:b0e45f0d:825fae73:fc8b148d
         Events : 0.1 www.6park.com
    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       34        1      active sync   /dev/sdc2
       2       8       35        2      active sync   /dev/sdc3
       3       8       37        3      active sync   /dev/sdc5
       5       8       38        4      spare rebuilding   /dev/sdc6 www.6park.com

--- Create partition on a raid device -------------------------- www.6park.com
Note: If you want to do this, I do not know what happen if you need
      to replace a disk. Not sure how to re-assemble the array when
      existing partition table is destroyed. I never tried this before. www.6park.com
      If you find out, post here and show me. www.6park.com
[root@vmvc01 ~]# fdisk /dev/md0
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable. www.6park.com

The number of cylinders for this disk is set to 24009024.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) www.6park.com
Command (m for help): p www.6park.com
Disk /dev/md0: 98.3 GB, 98340962304 bytes
2 heads, 4 sectors/track, 24009024 cylinders
Units = cylinders of 8 * 512 = 4096 bytes www.6park.com
    Device Boot      Start         End      Blocks   Id  System www.6park.com
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-24009024, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-24009024, default 24009024):
Using default value 24009024 www.6park.com
Command (m for help): p www.6park.com
Disk /dev/md0: 98.3 GB, 98340962304 bytes
2 heads, 4 sectors/track, 24009024 cylinders
Units = cylinders of 8 * 512 = 4096 bytes www.6park.com
    Device Boot      Start         End      Blocks   Id  System
/dev/md0p1               1    24009024    96036094   83  Linux www.6park.com
Command (m for help): w
The partition table has been altered! www.6park.com
Calling ioctl() to re-read partition table. www.6park.com
WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks. www.6park.com

--- Create FS on raid5 device ---------------------------------- www.6park.com
[root@vmvc01 ~]# mkfs -t ext3 -L raid5_00 /dev/md0
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
12009472 inodes, 24009024 blocks
1200451 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
733 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872 www.6park.com
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done www.6park.com
This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override. www.6park.com
[root@vmvc01 ~]# blkid
/dev/sda3: LABEL="/" UUID="c2830efe-e15d-445c-911e-91fea62a2117" TYPE="ext3"
/dev/sda2: LABEL="SWAP-sda2" TYPE="swap"
/dev/sda1: LABEL="/boot" UUID="2e850d99-c189-457f-b509-73ab217938a1" TYPE="ext3"
/dev/loop0: LABEL="RHEL/5.5 x86_64 DVD" TYPE="iso9660"
/dev/sdb1: LABEL="ORACLE_HOME" UUID="dbb254f6-2a05-4fe6-861e-d0e5614dc68c" TYPE="ext3"
/dev/sdc1: LABEL="raid5_00" UUID="803e59c9-3b97-4bf2-bc5b-caf8444950d0" SEC_TYPE="ext2" TYPE="ext3"
/dev/sdc6: LABEL="raid5_00" UUID="803e59c9-3b97-4bf2-bc5b-caf8444950d0" SEC_TYPE="ext2" TYPE="ext3"
/dev/md0: LABEL="raid5_00" UUID="803e59c9-3b97-4bf2-bc5b-caf8444950d0" SEC_TYPE="ext2" TYPE="ext3" www.6park.com
---- Mounting RAID5 -------------------------- www.6park.com
[root@vmvc01 ~]# more /etc/fstab
LABEL=/                 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sda2         swap                    swap    defaults        0 0 www.6park.com
# --- Oracle File System -------------------------------------------------- www.6park.com
LABEL=ORACLE_HOME       /d01                    ext3    defaults        1 2 www.6park.com
# --- RAID5 Oracle Data File System --------------------------------------- www.6park.com
LABEL=raid5_00          /d02                    ext3    defaults        1 2 www.6park.com
# --- NFS ----------------------------------------------------------------- www.6park.com
netfiler:/raid5_00      /raid5_00               nfs   timeo=14,intr,rw
netfiler:/raid5_01      /raid5_01               nfs   timeo=14,intr,rw
netfiler:/raid5_02      /raid5_02               nfs   timeo=14,intr,rw
netfiler:/raid5_03      /raid5_03               nfs   timeo=14,intr,rw www.6park.com

[root@vmvc01 ~]# mount -a
[root@vmvc01 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              51G  4.3G   44G   9% /
/dev/sda1              99M   15M   79M  16% /boot
tmpfs                 1.7G     0  1.7G   0% /dev/shm
/dev/hdc              362K  362K     0 100% /media/CDROM
/dev/hda              362K  362K     0 100% /media/CDROM_
netfiler:/raid5_00    939G  161G  731G  18% /raid5_00
netfiler:/raid5_01    939G   73G  819G   9% /raid5_01
netfiler:/raid5_02    939G   78G  814G   9% /raid5_02
netfiler:/raid5_03    852G   36G  773G   5% /raid5_03
/raid5_00/Software/Linux/Redhat/RedhatLinuxES5-Update5/rhel-5.5-server-x86_64-dvd.iso
                      3.5G  3.5G     0 100% /mnt/cdrom
/dev/sdb1              30G  173M   28G   1% /d01
/dev/md0               91G  188M   86G   1% /d02 www.6park.com

--- Create consistent raid5 config file and this is done ------------------------------------ www.6park.com
[root@vmvc01 ~]# mdadm --detail --scan >> /etc/mdadm.conf
[root@vmvc01 ~]# more /etc/mdadm.conf
ARRAY /dev/md0 level=raid5 num-devices=5 metadata=0.90 spares=1 UUID=cbe75ee2:b0e45f0d:825fae73:fc8b148d www.6park.com
--- Now, you can do -------------------------------------------------------- www.6park.com
1.) Failed a disk by marking it as failed to simulate a failure
2.) Stop The array
3.) Remove the failed disk
4.) Replace the failed disk
5.) Re-Assemble the array
6.) Check Resync status www.6park.com
Enjoy your hacking with Linux..... www.6park.com