How to increase the size of a linux lvm by expanding the virtual machine disk

How to increase the size of a linux lvm by expanding the virtual machine disk

Increasing the virtual hard disk

First off we increase the allocated disk space on the virtual machine itself. This is done by right clicking the virtual machine in vSphere, selecting edit settings, and then selecting the hard disk. In the below image I have changed the previously set hard disk of 20gb to 30gb while the virtual machine is up and running. Once complete click OK, this is all that needs to be done in VMware for this process.

If you are not able to modify the size of the disk, the provisioned size setting is greyed out. This can happen if the virtual machine has a snapshot in place, these will need to be removed prior to making the changes to the disk. Alternatively you may need to shut down the virtual machine if it does not allow you to add or increase disks on the fly, if this is the case make the change then power it back on.
Detect the new disk space

Once the physical disk has been increased at the hardware level, we need to get into the operating system and create a new partition that makes use of this space to proceed.

Before we can do this we need to check that the new unallocated disk space is detected by the server, you can use “fdisk -l” to list the primary disk. You will most likely see that the disk space is still showing as the same original size, at this point you can either reboot the server and it will detect the changes on boot or you can rescan your devices to avoid rebooting by running the below command. Note you may need to change host0 depending on your setup.

echo 1>/sys/block/sda/device/rescan

Below is an image after performing this and confirming that the new space is displaying.

Partition the new disk space

As outlined in my previous images the disk in my example that I am working with is /dev/sda, so we use fdisk to create a new primary partition to make use of the new expanded disk space. Note that we do not have 4 primary partitions already in place, making this method possible.

fdisk /dev/sda

We are now using fdisk to create a new partition, the inputs I have entered in are shown below in bold. Note that you can press ‘m’ to get a full listing of the fdisk commands.

‘n’ was selected for adding a new partition.

WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to
switch off the mode (command ‘c’) and change display units to
sectors (command ‘u’).

Command (m for help): n

‘p’ is then selected as we are making a primary partition.

Command action
l logical (5 or over)
p primary partition (1-4)
p

As I already have /dev/sda1 and /dev/sda2 as shown in previous images, I have gone with using ‘3’ for this new partition which will be created as /dev/sda3

Partition number (1-4): 3

We just press enter twice above as by default the first and last cylinders of the unallocated space should be correct. After this the partition is then ready.

First cylinder (2611-3916, default 2611): “enter”
Using default value 2611
Last cylinder, +cylinders or +size{K,M,G} (2611-3916, default 3916): “enter”
Using default value 3916

‘t’ is selected to change to a partition’s system ID, in this case we change to ‘3’ which is the one we just created.

Command (m for help): t
Partition number (1-5): 3

The hex code ‘8e’ was entered as this is the code for a Linux LVM which is what we want this partition to be, as we will be joining it with the original /dev/sda5 Linux LVM.

Hex code (type L to list codes): 8e
Changed system type of partition 3 to 8e (Linux LVM)

‘w’ is used to write the table to disk and exit, basically all the changes that have been done will be saved and then you will be exited from fdisk.

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

You will see a warning which basically means in order to use the new table with the changes a system reboot is required. If you can not see the new partition using “fdisk -l” you may be able to run “partprobe -s” to rescan the partitions. In my test I did not require either of those things at this stage (I do a reboot later on), straight after pressing ‘w’ in fdisk I was able to see the new /dev/sda3 partition of my 10gb of space as displayed in the below image.

For CentOS/RHEL run a “partx -a /dev/sda3” to avoid rebooting later on.

That’s all for partitioning, we now have a new partition which is making use of the previously unallocated disk space from the increase in VMware.
Increasing the logical volume

We use the pvcreate command which creates a physical volume for later use by the logical volume manager (LVM). In this case the physical volume will be our new /dev/sda3 partition.

[email protected]:~# pvcreate /dev/sda3
Device /dev/sda3 not found (or ignored by filtering).

In order to get around this you can either reboot, or use partprobe/partx as previously mentioned to avoid a reboot, as in this instance the disk does not appear to be there correctly despite showing in “fdisk -l”. After a reboot or partprobe/partx use the same command which will succeed.

[email protected]:~# pvcreate /dev/sda3
Physical volume “/dev/sda3” successfully created

Next we need to confirm the name of the current volume group using the vgdisplay command. The name will vary depending on your setup, for me it is the name of my test server. vgdisplay provides lots of information on the volume group, I have only shown the name and the current size of it for this example.

[email protected]:~# vgdisplay
— Volume group —
VG Name Mega

VG Size 19.76 GiB

Now we extend the ‘Mega’ volume group by adding in the physical volume of /dev/sda3 which we created using the pvcreate command earlier.

[email protected]:~# vgextend Mega /dev/sda3
Volume group “Mega” successfully extended

Using the pvscan command we scan all disks for physical volumes, this should confirm the original /dev/sda5 partition and the newly created physical volume /dev/sda3

[email protected]:~# pvscan
PV /dev/sda5 VG Mega lvm2 [19.76 GiB / 0 free]
PV /dev/sda3 VG Mega lvm2 [10.00 GiB / 10.00 GiB free]
Total: 2 [29.75 GiB] / in use: 2 [29.75 GiB] / in no VG: 0 [0 ]

Next we need to increase the logical volume (rather than the physical volume) which basically means we will be taking our original logical volume and extending it over our new partition/physical volume of /dev/sda3.

Firstly confirm the path of the logical volume using lvdisplay. This path name will vary depending on your setup.

[email protected]:~# lvdisplay
— Logical volume —
LV Path /dev/Mega/root

The logical volume is then extended using the lvextend command.

[email protected]:~# lvextend /dev/Mega/root /dev/sda3
Extending logical volume root to 28.90 GiB
Logical volume root successfully resized

There is then one final step which is to resize the file system so that it can take advantage of this additional space, this is done using the resize2fs command for ext based file systems. Note that this may take some time to complete, it took about 30 seconds for my additional space.

[email protected]:~# resize2fs /dev/Mega/root
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/Mega/root is mounted on /; on-line resizing required
old desc_blocks = 2, new_desc_blocks = 2
Performing an on-line resize of /dev/Mega/root to 7576576 (4k) blocks.
The filesystem on /dev/Mega/root is now 7576576 blocks long.

Alternatively if you’re running the XFS file system (default as of RedHat/CentOS 7) you can grow the file system with “xfs_growfs /dev/Mega/root”.

That’s it, now with the ‘df’ command we can see that the total available disk space has been increased.

Intel CPU EVC Matrix (VMware Enhanced vMotion Compatibility)

Intel CPU EVC Matrix (VMware Enhanced vMotion Compatibility)

Intel uses a model named “Tick-Tock” to follow every microarchitectural change with a die shrink. This results in having two EVC baselines for every microarchitecture. I’ve created a small table with a quick overview about EVC Modes with their appropriate CPU Series and Codenames used by Intel to denote their CPUs. I’ve also included additional Codenames that may be used by Intel for special processors. The upcoming Haswell architecture is named, but not yet available or supported by VMware.

 

EVC introduced Baseline µarch aka. CPU Series
L0 vSphere 4.0 Merom Core Conroe
Tigerton
Woodcrest
Clovertown
Kentsfield
30xx Series
32xx Series
51xx Series
53xx Series
72xx Series
73xx Series
L1 Penryn Dunnington
Harpertown
Yorkfield
Wolfdale
31xx Series
33xx Series
52xx Series
54xx Series
74xx Series
L2 Nehalem Nehalem Beckton
Gainestown
Bloomfield
Lynnfield
Clarksfield
34xx Lynnfield
35xx Series
55xx Series
65xx Series
75xx Series
i3-2100 Series
L3 vSphere 4.1 Westmere Gulftown
Clarkdale
Arrandale
i3/i5 Clarkdale
34xx Clarkdale
36xx Series
56xx Series
E7-2800 Series
E7-4800 Series
E7-8800 Seriee
Core i7-620LE
L4 vSphere 5.0 Sandy
Bridge
Sandy
Bridge
E3-1100 Series
E3-1200 Series
E5-1400 Series
E5-1600 Series
E5-2400 Series
E5-2600 Series
E5-4600 Series
L5 vSphere 5.1 Ivy Bridge i3-3200 Series
i7-3500-LE/UE Series
i7-3600-QE Series
Xeon E3-1100-C-v2 Series
Xeon E3-1200-v2 Series
Xeon E5-1400-v2 Series
Xeon E5-1600-v2 Series
Xeon E5-2400-v2 Series
Xeon E5-2600-v2 Series
Xeon E5-4600-v2 Series
Xeon E7-8800-v2 Series
Xeon E7-4800-v2 Series
Xeon E7-2800-v2 Series
L6  vSphere 6.0 Haswell Haswell E3-1200-v3 Series
E5-1400-v3 Series
E5-1600-v3 Series
E5-2400-v3 Series
E5-2600-v3 Series
i3-4300 Series
i5-4500-TE Series
i7-4700-EQ Series
L7  vSphere 6.5 Broadwell
Skylake Skylake

Intel CPU Series EVC Interoperability Matrix

EVC L0
Merom
EVC L1
Penryn
EVC L2
Nehalem
EVC L3
Westmere
EVC L4
Sandy Bridge
EVC L5
Ivy Bridge
Xeon 30xx Series
yes
no
no
no
no
no
Xeon 32xx Series
yes
no
no
no
no
no
Xeon 51xx Series
yes
no
no
no
no
no
Xeon 53xx Series
yes
no
no
no
no
no
Xeon 72xx Series
yes
no
no
no
no
no
Xeon 73xx Series
yes
no
no
no
no
no
Xeon 31xx Series
yes
yes
no
no
no
no
Xeon 33xx Series
yes
yes
no
no
no
no
Xeon 52xx Series
yes
yes
no
no
no
no
Xeon 54xx Series
yes
yes
no
no
no
no
Xeon 74xx Series
yes
yes
no
no
no
no
Xeon 34xx Lynnfield Series
yes
yes
yes
no
no
no
Xeon 35xx Series
yes
yes
yes
no
no
no
Xeon 55xx Series
yes
yes
yes
no
no
no
Xeon 65xx Series
yes
yes
yes
no
no
no
Xeon 75xx Series
yes
yes
yes
no
no
no
i3-2100 Series
yes
yes
yes
no
no
no
i3/i5 Clarkdale Series
yes
yes
yes
yes
no
no
Xeon 34xx Clarkdale Series
yes
yes
yes
yes
no
no
Xeon 36xx Series
yes
yes
yes
yes
no
no
Xeon 56xx Series
yes
yes
yes
yes
no
no
Xeon E7-2800 Series
yes
yes
yes
yes
no
no
Xeon E7-4800 Series
yes
yes
yes
yes
no
no
Xeon E7-8800 Series
yes
yes
yes
yes
no
no
Core i7-620LE
yes
yes
yes
yes
no
no
Xeon E3-1100 Series
yes
yes
yes
yes
yes
no
Xeon E3-1200 Series
yes
yes
yes
yes
yes
no
Xeon E5-1400 Series
yes
yes
yes
yes
yes
no
Xeon E5-1600 Series
yes
yes
yes
yes
yes
no
Xeon E5-2400 Series
yes
yes
yes
yes
yes
no
Xeon E5-2600 Series
yes
yes
yes
yes
yes
no
Xeon E5-4600 Series
yes
yes
yes
yes
yes
no
i7-3600-QE
yes
yes
yes
yes
yes
yes
Xeon E3-1100-C-v2 Series
yes
yes
yes
yes
yes
yes
Xeon E3-1200-v2 Series
yes
yes
yes
yes
yes
yes
Xeon E5-1400-v2 Series
yes
yes
yes
yes
yes
yes
Xeon E5-2400-v2 Series
yes
yes
yes
yes
yes
yes
Xeon E5-2600-v2 Series
yes
yes
yes
yes
yes
yes
Xeon E3-1200-v3 Series*
yes
yes
yes
yes
yes
yes
i3-4300 Series*
yes
yes
yes
yes
yes
yes
i5-4500-TE Series*
yes
yes
yes
yes
yes
yes
i7-4700-EQ Series*
yes
yes
yes
yes
yes
yes
*Haswell CPU EVC L0
Merom
EVC L1
Penryn
EVC L2
Nehalem
EVC L3
Westmere
EVC L4
Sandy Bridge
EVC L5
Ivy Bridge

Detailed CPU List

EVC Baseline CPU List
L0 Merom 3040, 3050, 3050, 3060, 3065, 3070, 5110, 5113, 5120, 5128, 5130, 5133, 5138, 5140, 5148, 5150, 5160, E5310, E5320, E5335, E5345, E7210, E7220, E7310, E7320, E7330, E7340, L5310, L5318, L5320, L5335, L7345, X3210, X3220, X3230, X5355, X5365, X7350
L1 Penryn E3110, E3120, E3120, E5205, E5220, E5405, E5410, E5420, E5430, E5440, E5450, E5462, E5472, E7420, E7430, E7440, E7450, L3110, L3360, L5215, L5240, L5410, L5420, L5430, L7445, L7455, X3320, X3330, X3350, X3360, X3370, X3380, X5260, X5270, X5272, X5450, X5460, X5470, X5472, X5482, X5492, X7460
L2 Nehalem E5502, E5503, E5503, E5504, E5506, E5507, E5520, E5530, E5540, E6510, E6540, E7520, E7530, E7540, L3426, L5506, L5520, L5530, L7545, L7555, W3520, W3530, W3540, W3550, W3565, W3570, W3580, W5580, W5590, X3430, X3440, X3450, X3460, X3470, X3480, X5550, X5560, X5570, X6550, X7542, X7550, X7560
L3 Westmere X5690, X5687, X5680, X5677, X5675, X5672, X5670, X5667, X5660, X5650, X5647, L5640, L5638, L5630, L5618, L5609, E5649, E5645, E5640, E5630, E5620, E5607, E5606, E5603, W3680, W3670, E7-8830, E7-8837, E7-8850, E7-8860, E7-8867L, E7-8870, E7-4807, E7-4820, E7-4830, E7-4850, E7-4860, E7-4870, E7-2803, E7-2820, E7-2830, E7-2850, E7-2860, E7-2870
L4 Sandy Bridge E3-1220, E3-1220L, E3-1225, E3-1230, E3-1235, E3-1240, E3-1245, E3-1260L, E3-1270, E3-1275, E3-1280, E3-1290, E5-1428L, E5-1620, E5-1650, E5-1660, E5-2403, E5-2407, E5-2418L, E5-2420, E5-2428L, E5-2430, E5-2430L, E5-2440, E5-2448L, E5-2450, E5-2450L, E5-2470, E5-2603, E5-2609, E5-2620, E5-2630, E5-2630L, E5-2637, E5-2640, E5-2643, E5-2648L, E5-2650, E5-2650L, E5-2658, E5-2660, E5-2665, E5-2667, E5-2670, E5-2680, E5-2687W, E5-2690, E5-4603, E5-4607, E5-4610, E5-4617, E5-4620, E5-4640, E5-4650, E5-4650L, G440, G460, G465, G470, G530, G530T, G540, G540T, G550, G550T, G555, G620, G620T, G622, G630, G630T, G632, G640, G640T, G645, G645T, G840, G850, G860, G860T, G870, i3-2100, i3-2100T, i3-2102, i3-2105, i3-2120, i3-2120T, i3-2125, i3-2130, i3-2310E, i3-2310M, i3-2312M, i3-2328M, i3-2330E, i3-2330M, i3-2340UE, i3-2348M, i3-2350M, i3-2357M, i3-2365M, i3-2367M, i3-2370M, i3-2375M, i3-2377M, i5-2300, i5-2310, i5-2320, i5-2380P, i5-2390T, i5-2400, i5-2400S, i5-2405S, i5-2410M, i5-2430M, i5-2435M, i5-2450M, i5-2450P, i5-2467M, i5-2500, i5-2500K, i5-2500S, i5-2500T, i5-2510E, i5-2515E, i5-2520M, i5-2537M, i5-2540M, i5-2550K, i5-2557M, i7-2600, i7-2600K, i7-2600S, i7-2610UE, i7-2617M, i7-2620M, i7-2629M, i7-2630QM, i7-2635QM, i7-2637M, i7-2640M, i7-2649M, i7-2655LE, i7-2657M, i7-2670QM, i7-2675QM, i7-2677M, i7-2700K, i7-2710QE, i7-2715QE, i7-2720QM, i7-2760QM, i7-2820QM, i7-2860QM, i7-2920XM, i7-2960XM, i7-3820, i7-3930K, i7-3960X, i7-3970X, 807UE, 827E, 847E, B710, B720, B800, B810, B810E, B815, B820, B830, B840, B940, B950, B960, B970, B980
L5 Ivy Bridge E3-1220LV2, E3-1220V2, E3-1225V2, E3-1230V2, E3-1240V2, E3-1245V2, E3-1265LV2, E3-1270V2, E3-1275V2, E3-1280V2, E3-1290V2, E5-1428LV2, E5-1620V2, E5-1650V2, E5-1660V2, E5-2403V2, E5-2407V2, E5-2418LV2, E5-2420V2, E5-2428LV2, E5-2430LV2, E5-2430V2, E5-2440V2, E5-2448LV2, E5-2450LV2, E5-2450V2, E5-2470V2, E5-2603V2, E5-2609V2, E5-2618LV2, E5-2620V2, E5-2628LV2, E5-2630LV2, E5-2630V2, E5-2637V2, E5-2640V2, E5-2643V2, E5-2648LV2, E5-2650LV2, E5-2650V2, E5-2658V2, E5-2660V2, E5-2667V2, E5-2670V2, E5-2680V2, E5-2687WV2, E5-2690V2, E5-2695V2, E5-2697V2, E5-4603V2, E5-4607V2, E5-4610V2, E5-4620V2, E5-4624LV2, E5-4627V2, E5-4640V2, E5-4650V2, E5-4657LV2, E7-2850V2, E7-2870V2, E7-2880V2, E7-2890V2, E7-4809V2, E7-4820V2, E7-4830V2, E7-4850V2, E7-4860V2, E7-4870V2, E7-4880V2, E7-4890V2, E7-8850V2, E7-8857V2, E7-8870V2, E7-8880LV2, E7-8880V2, E7-8890V2, E7-8891V2, E7-8893V2, G1610, G1610T, G1620, G1620T, G1630, G2010, G2020, G2020T, G2030, G2030T, G2100T, G2120, G2120T, G2130, G2140, i3-3110M, i3-3120M, i3-3120ME, i3-3130M, i3-3210, i3-3217U, i3-3217UE, i3-3220, i3-3220T, i3-3225, i3-3227U, i3-3229Y, i3-3240, i3-3240T, i3-3245, i3-3250, i3-3250T, i5-3210M, i5-3210M, i5-3230M, i5-3230M, i5-3317U, i5-3320M, i5-3330, i5-3330S, i5-3337U, i5-3339Y, i5-3340, i5-3340M, i5-3340S, i5-3350P, i5-3360M, i5-3380M, i5-3427U, i5-3437U, i5-3439Y, i5-3450, i5-3450S, i5-3470, i5-3470S, i5-3470T, i5-3475S, i5-3550, i5-3550S, i5-3570, i5-3570K, i5-3570S, i5-3570T, i5-3610ME, i7-3517U, i7-3517UE, i7-3520M, i7-3537U, i7-3540M, i7-3555LE, i7-3610QE, i7-3610QM, i7-3612QE, i7-3612QM, i7-3612QM, i7-3615QE, i7-3615QM, i7-3630QM, i7-3632QM, i7-3632QM, i7-3635QM, i7-3667U, i7-3687U, i7-3689Y, i7-3720QM, i7-3740QM, i7-3770, i7-3770K, i7-3770S, i7-3770T, i7-3820QM, i7-3840QM, i7-3920XM, i7-3940XM, i7-4820K, i7-4930K, i7-4960X, 1000M, 1005M, 1007U, 1017U, 1019Y, 1020E, 1020M, 1037U, 1047UE, 1405V2, 2020M, 2030M, 2117U, 2127U, 2129Y, 927UE, A1018
L6 Haswell E3-1220 v3, E3-1220LV3, E3-1225V3, E3-1226V3, E3-1230 v3, E3-1230Lv3, E3-1231V3, E3-1240 v3, E3-1240LV3, E3-1241V3, E3-1245 v3, E3-1246V3, E3-1265Lv3, E3-1268LV3, E3-1270 v3, E3-1271V3, E3-1275 v3, E3-1275LV3, E3-1276V3, E3-1280 v3, E3-1281V3, E3-1285 v3, E3-1285Lv3, E3-1286LV3, E3-1286V3, E5-1428LV3, E5-1620V3, E5-1630V3, E5-1650V3, E5-1660V3, E5-1680V3, E5-2408LV3, E5-2418LV3, E5-2428LV3, E5-2438LV3, E5-2603V3, E5-2608LV3, E5-2609V3, E5-2618LV3, E5-2620V3, E5-2623V3, E5-2628LV3, E5-2630LV3, E5-2630V3, E5-2637V3, E5-2640V3, E5-2643V3, E5-2648LV3, E5-2650LV3, E5-2650V3, E5-2658AV3, E5-2658V3, E5-2660V3, E5-2667V3, E5-2670V3, E5-2680V3, E5-2683V3, E5-2687WV3, E5-2690V3, E5-2695V3, E5-2697V3, E5-2698V3, E5-2699V3, G1820, G1820T, G1820TE, G1830, G1840, G1840T, G1850, G3220, G3220T, G3240, G3240T, G3250, G3250T, G3258, G3320TE, G3420, G3420T, G3430, G3440, G3440T, G3450, G3450T, G3460, i3-4000M, i3-4005U, i3-4010U, i3-4010Y, i3-4012Y, i3-4020Y, i3-4025U, i3-4030U, i3-4030Y, i3-4100E, i3-4100M, i3-4100U, i3-4102E, i3-4110E, i3-4110M, i3-4112E, i3-4120U, i3-4130, i3-4130T, i3-4150, i3-4150T, i3-4158U, i3-4160, i3-4160T, i3-4330, i3-4330T, i3-4330TE, i3-4340, i3-4340TE, i3-4350, i3-4350T, i3-4360, i3-4360T, i3-4370, i5-4200H, i5-4200M, i5-4200U, i5-4200Y, i5-4202Y, i5-4210H, i5-4210M, i5-4210U, i5-4210Y, i5-4220Y, i5-4250U, i5-4258U, i5-4260U, i5-4278U, i5-4288U, i5-4300M, i5-4300U, i5-4300Y, i5-4302Y, i5-4308U, i5-4310M, i5-4310U, i5-4330M, i5-4340M, i5-4350U, i5-4360U, i5-4400E, i5-4402E, i5-4402EC, i5-4410E, i5-4422E, i5-4430, i5-4430S, i5-4440, i5-4440S, i5-4460, i5-4460S, i5-4460T, i5-4570, i5-4570S, i5-4570T, i5-4570TE, i5-4590, i5-4590S, i5-4590T, i5-4670, i5-4670K, i5-4670S, i5-4670T, i5-4690, i5-4690S, i5-4690T, i7-4500U, i7-4510U, i7-4550U, i7-4558U, i7-4578U, i7-4600M, i7-4600U, i7-4610M, i7-4610Y, i7-4650U, i7-4700EC, i7-4700EQ, i7-4700HQ, i7-4700MQ, i7-4702EC, i7-4702HQ, i7-4702MQ, i7-4710HQ, i7-4710MQ, i7-4712HQ, i7-4712MQ, i7-4720HQ, i7-4722HQ, i7-4765T, i7-4770, i7-4770K, i7-4770S, i7-4770T, i7-4770TE, i7-4771, i7-4785T, i7-4790, i7-4790S, i7-4790T, i7-4800MQ, i7-4810MQ, i7-4900MQ, i7-4910MQ, i7-4930MX, i7-4940MX, i7-5820K, i7-5930K, i7-5960X, 2000E, 2002E, 2950M, 2955U, 2957U, 2961Y, 2970M, 2980U, 2981U, 3550M, 3556U, 3558U, 3560M, 3560Y, 3561Y
? Broadwell i7-5557U, i7-5550U, i7-5500U, i7-5650U, i7-5600U, i5-5287U, i5-5257U, i5-5250U, i5-5200U, i5-5350U, i5-5300U, i3-5157U, i3-5010U, i3-5005U, 3805U, 3755U, 3205U, 5Y71, 5Y70, 5Y51, 5Y31, 5Y10c, 5Y10a, 5Y10
Mysql DB size

Mysql DB size

With the following query you will get summary of all DB size

SELECT table_schema AS "Database",
ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS "Size (MB)"
FROM information_schema.TABLES
GROUP BY table_schema;

You can also explode single DB and get size of each table with the following query (change db_name with the name of database you want to analize)

SELECT table_name AS "Table",
ROUND(((data_length + index_length) / 1024 / 1024), 2) AS "Size (MB)"
FROM information_schema.TABLES
WHERE table_schema = "db_name"
ORDER BY (data_length + index_length) DESC;

Otherwise you can get size of table on all database sort by size with the following queyr

SELECT 
     table_schema as `Database`, 
     table_name AS `Table`, 
     round(((data_length + index_length) / 1024 / 1024), 2) `Size in MB` 
FROM information_schema.TABLES 
ORDER BY (data_length + index_length) DESC;
How to use the DNS translation feature

How to use the DNS translation feature

Description
The DNS translation feature available in the FortiOS firmware is designed to modify the DNS reply from a DNS server.

It is typically used to allow internal users of a network to access resources with their private IP addresses, hence can simplify the firewall configurations.

A network diagram is provided below with an example that illustrates on how to configure this feature.

In this example, the client sends a DNS resolution request to the DNS server 172.31.17.252 for resource “server1.lab.mycompany.com” . The DNS reply sent by the DNS server is 172.31.17.37 (this is the public IP address of “server1”), but the reply is translated on the FortiGate unit into 10.73.1.37, which is the private IP address of the same resource, “server1”.

 

Solution
How internal users can access internal resources via an external VIP

How internal users can access internal resources via an external VIP

Products
FortiGate
Description
This article describes a solution for the following requirement :

A user located to an internal LAN needs to access a server located on an internal LAN or DMZ by using however a public Virtual IP on the Fortigate.
External users also access the same server via the “external” port.

The following diagram illustrates this scenario :

port3
10.67.2.82 10.67.0.176 port1
[ INTERNAL USER ] ====|==== [ FortiGate ] == VIP 172.31.16.164 == [ EXTERNAL USERS ]
| | 172.31.19.113
[ SERVER1 ] ==========| | port2
10.67.0.164 |10.67.0.176
|
[ SERVER2 ] ==========================|
10.91.3.113

The INTERNAL USER PC is accessing the SERVER1 and SERVER2 with destination IP = 172.31.16.164 or 172.31.19.113, which in turn gets translated to the real servers IP and routed back to the servers.
Solution
FortiGate configuration excerpt :
========================

config firewall vip
edit “SERVER1”
set extip 172.31.16.164
set extintf “any” <<< Specifying “any” is a requirement
set mappedip 10.67.0.164
next
end
edit “SERVER2”
set extip 172.31.19.113
set extintf “any” <<< Specifying “any” is a requirement set mappedip 10.91.3.113 next config firewall policy edit 4 set srcintf “port1” set dstintf “port3” set srcaddr “all” set dstaddr “SERVER1” set action accept set schedule “always” set service “ANY” next edit 3 set srcintf “port3” set dstintf “port3” set srcaddr “all” set dstaddr “SERVER1” set action accept set schedule “always” set service “ANY” next edit 5 set srcintf “port1” set dstintf “port2” set srcaddr “all” set dstaddr “SERVER2” set action accept set schedule “always” set service “ANY” next edit 6 set srcintf “port3” set dstintf “port2” set srcaddr “all” set dstaddr “SERVER2” set action accept set schedule “always” set service “ANY” next end Note : policy 4 and 5 are used for external users ; policy 3 and 6 are used for internal users. Traffic flow snippet an HTTP session from USER to SERVER1: ============================================== FGT-1 (root) # diagnose sniffer packet any “host 10.67.2.82 or host 10.67.0.164” 4 interfaces=[any] filters=[host 10.67.2.82 or host 10.67.0.164] 6.798488 port3 in 10.67.2.82.2080 -> 172.31.16.164.8090: syn 391946722
6.798556 port3 out 10.67.0.176.26756 -> 10.67.0.164.80: syn 391946722
6.798856 port3 in 10.67.0.164.80 -> 10.67.0.176.26756: syn 3548167716 ack 391946723
6.798873 port3 out 172.31.16.164.8090 -> 10.67.2.82.2080: syn 3548167716 ack 391946723
6.799125 port3 in 10.67.2.82.2080 -> 172.31.16.164.8090: ack 3548167717
6.799131 port3 out 10.67.0.176.26756 -> 10.67.0.164.80: ack 3548167717

Note : for this traffic (port3 to port3), even though NAT is not enabled on the policy, the source IP address gets translated with the Fortigate internal IP address.

Session list for an HTTP session from USER to SERVER1
==========================================

FGT-1 (root) # diagnose sys session list

session info: proto=6 proto_state=05 duration=2 expire=0 timeout=3600 flags=00000000 sockflag=00000000 sockport=0 av_idx=0 use=5
origin-shaper=
reply-shaper=
per_ip_shaper=
ha_id=0 hakey=758
policy_dir=0 tunnel=/
state=may_dirty
statistic(bytes/packets/allow_err): org=3407/13/1 reply=18467/19/1 tuples=4
orgin->sink: org pre->post, reply pre->post dev=4->4/4->4 gwy=10.67.0.164/10.67.2.82
hook=pre dir=org act=dnat 10.67.2.82:2075->172.31.16.164:8090(10.67.0.164:80)
hook=post dir=org act=snat 10.67.2.82:2075->10.67.0.164:80(10.67.0.176:26815)
hook=pre dir=reply act=dnat 10.67.0.164:80->10.67.0.176:26815(10.67.2.82:2075)
hook=post dir=reply act=snat 10.67.0.164:80->10.67.2.82:2075(172.31.16.164:8090)
pos/(before,after) 0/(0,0), 0/(0,0)
misc=0 policy_id=3 id_policy_id=0 auth_info=0 chk_client_info=0 vd=0
serial=0080d305 tos=ff/ff ips_view=0 app_list=0 app=0
dd_type=0 dd_rule_id=0
per_ip_bandwidth meter: addr=10.67.2.82, bps=19399

Traffic flow snippet an HTTP session from USER to SERVER2:
==============================================

FGT1# diagnose sniffer packet any “host 10.67.2.82 or host 10.91.3.113 and port 80” 4

4.741440 port3 in 10.67.2.82.2726 -> 172.31.19.113.80: syn 53278201
4.741515 port2 out 10.67.2.82.2726 -> 10.91.3.113.80: syn 53278201
4.741697 port2 in 10.91.3.113.80 -> 10.67.2.82.2726: syn 153837872 ack 53278202
4.741722 port3 out 172.31.19.113.80 -> 10.67.2.82.2726: syn 153837872 ack 53278202
4.742024 port3 in 10.67.2.82.2726 -> 172.31.19.113.80: ack 153837873
4.742033 port2 out 10.67.2.82.2726 -> 10.91.3.113.80: ack 153837873
4.742917 port3 in 10.67.2.82.2726 -> 172.31.19.113.80: psh 53278202 ack 153837873
4.742924 port2 out 10.67.2.82.2726 -> 10.91.3.113.80: psh 53278202 ack 153837873
4.743306 port2 in 10.91.3.113.80 -> 10.67.2.82.2726: ack 53279042
4.743315 port3 out 172.31.19.113.80 -> 10.67.2.82.2726: ack 53279042

Session list for an HTTP session from USER to SERVER2
==========================================

session info: proto=6 proto_state=01 duration=2 expire=3597 timeout=3600 flags=00000000 sockflag=00000000 sockport=0 av_idx=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
ha_id=0 hakey=1475
policy_dir=0 tunnel=/
state=may_dirty
statistic(bytes/packets/allow_err): org=92/2/1 reply=52/1/1 tuples=2
orgin->sink: org pre->post, reply pre->post dev=4->3/3->4 gwy=10.91.3.113/10.67.2.82
hook=pre dir=org act=dnat 10.67.2.82:2752->172.31.19.113:80(10.91.3.113:80)
hook=post dir=reply act=snat 10.91.3.113:80->10.67.2.82:2752(172.31.19.113:80)
pos/(before,after) 0/(0,0), 0/(0,0)
misc=0 policy_id=6 id_policy_id=0 auth_info=0 chk_client_info=0 vd=0
serial=0082338c tos=ff/ff ips_view=0 app_list=0 app=0
dd_type=0 dd_rule_id=0
per_ip_bandwidth meter: addr=10.67.2.82, bps=4049

Fixing Office 365 DirSync account matching issues

Fixing Office 365 DirSync account matching issues

DirSyncServerRecently I had to fix some issues with DirSync. For some reason (there were some cloud users created before DirSync was enabled) there were duplicate users, because DirSync failed to match the already present cloud user and the corresponding AD (Active Directory) user. There were also accounts that failed to sync and thus failed to sync all attributes properly.

If there is already a cloud account and there is need for a synced account, you can create an AD account in DirSynced OU’s. But be sure to create the user with a full UPN matching the one in Office 365 and SMTP addresses that are present on the Cloud account. With the next sync it should match both accounts. If not, it fails matching and you end up with either duplicate accounts (one cloud user and a DirSynced user with the same name/lastname/displayname) or get an InvalidSoftMatch.

When UPN/SMTP matching failed you can merge those accounts again by setting the ImmutableID on the Office 365 account (MsolUser) which is derived from the AD user’s ObjectGuid. You can only add this attribute to Office 365 accounts. After this is set, DirSync should match the accounts correctly.

So, how did I resolve this? See below:

When there are duplicates:

    • Remove user from DirSync (move to OU which is not synced, will only work when OU Filtering is used. If not, disable DirSync…).
    • Perform DirSync.
    • Remove duplicate synced user (NOT cloud user):
      • Remove-MSOLuser -UserPrincipalName <UPN> -RemoveFromRecycleBin
      • Add ImmutableID from AD user to Cloud user
        • $guid = (get-Aduser <username>).ObjectGuid
          $immutableID = [System.Convert]::ToBase64String($guid.tobytearray())
        • Connect to AD Azure (Connect-MSOLService when AD Azure Powershell Module is installed).
        • Set-MSOLuser -UserPrincipalName <clouduserUPN> -ImmutableID $immutableID
        • It’s possible that the clouduserUPN must be changed to the <tenant>.onmicrosoft.com format. It should be changed by DirSync to correspond with the AD UPN.
        • See also http://www.joseph-streeter.com/?p=423
    • Place account back in correct (synced) AD OU.
    • Manually kick off a sync on the DirSync Server if you don’t want to wait (up to 3 hours with default settings):
      • C:\Program Files\Windows Azure Directory Sync\DirSyncConfigShell.psc1
      • Start-OnlineCoexistenceSync

In my case it didn’t always match the accounts and was required to perform a Full DirSync (on DirSync server):

    • Via MIISClient, Management Agents:
      • C:\Program Files\Windows Azure Active Directory Sync\SYNCBUS\Synchronization Service\UIShell\missclient.exe
    • Be sure to be member of local group “FIMSyncAdmins”
      Names might be different depending on DirSync version
    • On the Windows Azure Active Directory Connector:
      • Properties>Run>Full Import Delta Sync
    • on the Active Directory Connector:
      • Properties>Run>Full Import Full Sync
  • Note that a Full Sync can take a long time if you have a lot of objects. Furthermore, changes can take a while to propagate in Office 365.
  • It might be necessary to edit an attribute (Description, office etc. Something that is synced), and then perform a (normal) sync.

When you have an InvalidSoftMatch (SMTP Address matching doesn’t work because SMTP address already exists in Cloud):

Within the MIISClient.exe on the DirSync server, you can check for errors. In this case the account wasn’t properly matched:

clip_image001

  • Add ImmutableID from AD user to Cloud user:
    • $guid = (get-Aduser <username>).ObjectGuid
    • $immutableID = [System.Convert]::ToBase64String($guid.tobytearray())
    • Connect to AD Azure (Connect-MSOLService when AD Azure Powershell Module is installed)
    • Set-MSOLuser -UserPrincipalName <clouduserUPN> -ImmutableID $immutableID
    • It’s possible that the clouduserUPN must be changed to the <tenant>.onmicrosoft.com format. It should be changed by DirSync to correspond with the AD UPN.
    • See also http://www.joseph-streeter.com/?p=423
  • Then perform a sync as described in the previous section.

In my case these procedures resolved my issues. But as always, use this information at your own risk. Best to make sure that you don’t end up in a situation like this Winking smile

See also:

One or more objects don’t sync when using the Azure Active Directory Sync tool http://support.microsoft.com/kb/2643629/en-us

How to use SMTP matching to match on-premises user accounts to Office 365 user accounts for directory synchronization http://support.microsoft.com/kb/2641663/en-us

Create a MySQL Slave using Replication with No Downtime

Create a MySQL Slave using Replication with No Downtime

2013060316014742_1

I have a customer who has over 100GB of MySQL data and taking their site down for even a few minutes is not feasible. I really wanted to get a slave set up in case the main server ever dies. Even though the server is backed up, it would take 2-3 hours (or longer) to restore the MySQL server which is not very acceptable.

The solution is to use replication. The traditional problem with this approach is locking the tables for so long while the mysqldump happens… for a database this size, close to 4-5 hours.

Idera’s Free Tool called Linux Hot Copy (hcp) was the answer I was looking for. By using hcp, you can lock the tables, make a near instant “snapshot”, record the master position, and unlock the tables. At your leisure, just copy the snapshot of the mysql data to your slave device, and start up your replication! This makes setting up new slaves a snap with minimal impact on your business.

First off, I will assume you have a production MySQL server in use and running. In my scenario, I am using CentOS 5.6 64Bit and MySQL 5.5. This tutorial will probably will work for older versions as well. I also will assume you know how to edit and copy files at the linux command line. If you don’t, you probably should get help from an experienced system administrator.

If you have not done so already, set up another mysql server for your slave. It should be a decent server, equal to your current live production server so you can switch to it in the event of failure.

I will also assume:

master server = 192.168.1.100
slave server = 192.168.2.200

You’ll need to substitute your IP Addressess in place of mine.

On Master Server (192.168.1.100):

1. Install Linux Hot Copy. Linux Hot Copy. If you need help with installation, here’s some documentation

2. Setup your Server ID and enable bin-logs. Note that bin logs record every change to your database, so make sure you have ample space to continue!)

Edit your /etc/my.cnf file and put these lines at the top, just under the [mysqld] line.

# enable mysql bin logs and server-id for mysql replication
       log-bin=mysql-bin
       server-id=1

Restart MySQL so bin logs are started. e.g. /etc/init.d/mysql restart you can verify it’s working by issuing the show master statusG command.

3. Create a user that has replication privs on the Master Server.

mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'192.168.2.200' IDENTIFIED BY 'password';

4. The next few steps will need to be done quickly so that you minimize your mysql server’s downtime. Make sure you know up-front the device (e.g. /dev/sda2) where your MySQL installation is located (typically /var/lib/mysql on CentOS):

Lock your Master MySQL Tables and show the status location of the bin log….

mysql> FLUSH TABLES WITH READ LOCK; SHOW MASTER STATUS;

Make sure you record and copy the information down, e.g. the filename and log position

From the command line, enter the following command, replacing /dev/sda2 with your raw device:

hcp /mnt/snap /dev/sda2

Back to MySQL, unlock your tables:

mysql> unlock tables;

Now you have a perfect copy of your “frozen” data at the following location: (may vary)..

FROZEN DATA LOCATION:
/mnt/snap

On Slave Server: 192.168.2.200

On the slave server, make sure MySQL is stopped and move the old mysql folder: (make sure this is the SLAVE SERVER 192.168.2.200 and NOT the live server!):

/etc/init.d/mysql stop
mv /var/lib/mysql /var/lib/mysql.old

Back on the Master Server: 192.168.1.100

1. Copy the “frozen” mysql data:

rsync -avz /mnt/snap  [email protected]:/var/lib/mysql

2. Copy my.cnf to slave:

scp /etc/my.cnf [email protected]_or_host:/etc/my.cnf

3. Once the Copy is Complete you can delete your “hot copy”

hcp -r /dev/hcp1

Now, go to your Slave Server: 192.168.2.200

1. edit /etc/my.cnf and change server-id to 2 and comment out or delete the log-bin line you added from the master..

2. start up mysql, and then enter commands to connect to master.. replacing the log file and position number with the ones you recorded earlier:

mysql> CHANGE MASTER TO 
      MASTER_HOST='192.168.1.100', 
      MASTER_USER='repl', 
      MASTER_PASSWORD='password', 
      MASTER_LOG_FILE='mysql-bin.000001',
      MASTER_LOG_POS=12345678;
mysql> START SLAVE;

 mysql> SHOW SLAVE STATUS/G;

MySQL will show how far it’s behind, it might take a few minutes to catch up depending on the number of changes that happened to your database during the copy.

I hope you enjoyed this tutorial on MySQL Replication with no downtime. Now it’s easy!

Linux shadow copy

Linux shadow copy

How to add shadowcopy to linux.

There’s a software that add shadowcopy to linux; below are instruction for Centos 6

First of all you must download and install the two package below

idera-hotcopy-5-14-4-x86_64

r1soft-setup-5-14-4-x86_64

Than you must install kernel-headers and kernel-devel for your running version.

If you’re running latest version you can use

yum install kenel-header kernel-devel

otherwise you can install directly from vault.centos.org repository as below

rpm -ivh http://vault.centos.org/6.6/updates/x86_64/Packages/kernel-devel-2.6.32-504.1.3.el6.x86_64.rpm
rpm -ivh http://vault.centos.org/6.6/updates/x86_64/Packages/kernel-headers-2.6.32-504.1.3.el6.x86_64.rpm

Now that you have software e kernel-headers you build your specific kernel-module for hcp device.

launch r1soft-setup-old –get-module to build it

now you can try to create shadow copy with

hcp /mnt/snapshot /dev/sda1

this will create a shadowcopy of /dev/sda1 on /mnt/snapshot