Beginners Guide to LVM 1 What is LVM? LVM stands for ``Logical Volume Manager''. Put in a few words, it is a flexible method to manage your disk space. This is a VERY simplistic statement! Let me take some words more. LVM allows you to overcome the space limitations of hard disks. With LVM it is easily possible to create filesystems spanning more than one disk without hassle. This is impossible with the disk partition schemes currently in use on most vanilla PC Linux systems. With LVM and help from its friends MD and reiserfs as well as ext2fsadm it is also possible to create and manage efficiently large amounts of disk space, create filesystems, mount them, resize them online (extending and shrinking), move data pysically from one disk to another without changing the position in the file tree as this would be done by other utilities and much more. And all this in a convenient manner. LVM allows you to handle disk space dynamically instead statically - that's at the heart of it. 2 Why LVM? Why would someone try to do things like these? They seem a little bit unusual, at least at the first moment. This might be true if your machine is a typical home machine: a box with one or two disks. While there are possibly more home machines out there than machines in data centers, the latter ones use special software or hardware to handle large amounts of disks. Imagine the situation a database machine, a seek engine or a portal is facing: large and steadily growing amounts of data produced by applications. If they offer some service like news or free email they will have enormously growing data sets. Now imagine yourself as administrator of such a machine, facing the threat of filesystems constantly filling up. Will you catch every filesystem before the dreaded message ``could not write to disk. Possibly filesystem full.'' occurs? Even if you were doing nothing than partitioning, formatting and adding disks you would hardly keep up with your work ... Lets say you bought a bunch of those fancy new 72GB disks, partition them in the usual way and put them into your ever space hungry server. You might feel safe for a moment, but what if a filesystem would grow bigger than that? Previously you had two choices: either knowing in advance which filesystems will fill up and partition your disks that way - or waiting until a filesystem grows nearly full, then copy the data out to tape or disk, destroy the file system and create a bigger new one, then copy the data back. And all of the time your data are not available for applications; if you add more disks, even your server has to be brought down (at least if you use SCSI or IDE disks, and thats most of us.). Of course this problem is not new and there have been numerous attempts to solve it: concatenated disks, RAID systems with special controllers and hot-swapable hardware and so on. Nothing wrong with these wonderful devices, you can use them with LVM as well. But they all complicate the handling of data much more than LVM does since they don't overcome the usual very low number of partitions. 3 Some Basic Concepts When booting, the BIOS of a PC looks at the first sector of a disk (the Master Boot Record, MBR) and validates that a boot record is found. Then control is passed to the boot loader in this structure which in turn reads more blocks from the disk, using the BIOS I/O routines. Later on, the OS takes control and handles I/O much better than the BIOS does, but in the crucial two first steps only the BIOS is involved. This is the reason for partitioning - the ability of the BIOS to read the blocks and BIOS routines define the layout of hard disks. Current disks on PCs can be partitioned with primary partitions and additonally extended partitions. A minimum of one primary partition is required and a maximum of four primary partitions per disk is allowed. As a method to handle disk partitions more flexible there is also the option to create one extended partition per disk at the cost of losing one primary partition. The extended partition itself is unusable for file systems. Instead it is used as a container to overcome the limitation of primary partitions: within an extended partition logical partitions can be created which in turn can be used. The BIOS normally requires that at least one primary partition is present which is tagged active and contains the OS (This is true at least for older operating systems - Linux as a clever OS has a number of boot managers which are able to boot the kernel even from logical partitions). Thus, many operating systems tend to configure the disk as one partition and use its space for all purposes. Not to mention multi-boot configurations, this solution is somewhat suboptimal: An operating system has to perform many different tasks, requiring different use of resources - disk space in our case. There is e.g. swap as well as storage for temporary files, applications, and there can be numerous more, depending on the type of system. All of these should be handled spearately, primarily to avoid data loss which could occur if there were only one file system and this would be damaged but also to gain a finer level of control e.g. to spread load across disks. Therefore often disks are configured with primary and extended partitions, and partitioning schemes for various purposes are a part of virtually every Linux installation guide. This topic also leads to lengthy debates since all of these schemes share a common problem: they can't be changed after installation without massive impact on the operating system, usually requiring copying around huge amounts of data or, worse, a fresh installation of the OS. This is where LVM comes into play. It allows to handle disk space very flexible not only at installation time but also anytime later. But for this we have to adopt some new ideas. Basically the BIOS limitation of four primary partitions can't be overcome but in the effect this is what happens. We will now have a look at the terms of LVM. 3.1 The overall picture 3.1.1 Physical Volumes A Physical Volume (PV) is the basic bulding block. It is a primary partition on a disk, usually only one out of the four possible. It is marked in the partition table with the label ``0x8e'' to avoid usage by other operating systems and to show that it is something special. One or more (up to 255) Physical Volumes form a Volume Group (VG). (simple picture of a disk with one partition) 3.1.2 Volume Groups Volume Groups are ``containers'' of disk space. Any definition of file systems, management of disk space and so on is done on their basis. Therefore some more informations are needed at this point. Imagine VGs as a pool of disk space, or, if you like, as an extensible, huge stock of storage where you don't have to buy all of the goods at one time. Insted you buy what you need for your daily or weekly subsistence with exact knowledge of the filling status of the stock. As long as there is some storage you can easily ``buy'' it and give it to the storage hungry file systems. And if your stock is running low you can add some more storage by adding disk(s) and give it to the file systems without the need to repartition or creating new mount points or to stop at a size where a normal disk would be full. Also you can react very dynamically to storage needs which weren't foreseeable at the moment the machine was set up. Up to 99 VGs are possible per machine - at least at the moment. This gives you the ability to manage 99(VG) x 255(PV) = 25245 disks per machine. (picture of a VG consisting of three disks) 3.1.3 Logical Volumes Now, this is the place where we can draw a comparison to the traditional storage scheme. Like partitions, Logical Volumes (LV) can contain file systems or can be used directly by applications which can handle ``raw partitions'' like databases. A maximum of 255 Logical Volumes can be defined per VG. Logical volumes can reside anywhere in a Volume Group, but they can't span the borders of a VG. This means, that a LV can be located on one or more disks - in fact, on many disks as are in the VG. And as well parts of the disk space of a PV can be allocated to more than one LV. The following picture tries to show this: (picture of the three disks in the VG, a LV spreading two of them) 3.2 The inner workings 3.2.1 Physical Entries To accomplish that goal the way chosen was to cut the PVs to pieces called Physical Entries (PE). The size of a PE is specific to the Volume Group, and all PVs in a VG have PEs of the same size. PEs are located on the disk - they are the smallest unit of disk space we are dealing with. The PE size is defined at the creation time of a VG and cannot be changed later. PEs can range in size from a few KBs up to several GBs. There is no standard size for a PE but a good measure is 4 to 8MB. PEs are numbered consequently on a PV, but not in a VG. There is a maximum of 65534 PEs per PV. (picture of a disk, showing the PEs) 3.2.2 Logical Entries A Logical Entry (LE) is the counterpart of and mapped to a Physical Entry. They are the building blocks of LVs much like PEs are the building blocks of PVs. Logical Volumes are formed by adding LEs together and mapping them to their respective PEs. This way the size of LVs can only be specified in powers of the size of a LE; also extensions of an LV are done in powers of an LE, and even if only one byte more would be needed the LV must be extended by one LE. LEs are numbered consequently in a LV, but not on a PV nor in the VG. Up to 65534 LEs are allowed per LV. (picture of a LV mapping its LEs to PEs on the disk) 4 O.K., and now? Installation! To begin to use LVM, you have to have LVM support in your kernel and the user space tools compiled and ready for use. There are some Linux distributions out there with LVM support already built in, e.g. SuSE6.3 (although that has a nasty bug with installation), but if you own an older Linux installation chances are that you have to download the sources and install it by yourself. LVM is available for kernels from 2.0.3x(???) on and is official part of the kernel since 2.3.47. The newest kernel patches are always found on http://linux.msede.com/lvm as well as older patches and a lot of more documentation.There is also a mailing list archive for LVM on http://linux.msede.com/lvm/mlist/archive. 4.1 Download, compilation and disk space Download the appropriate patch for you kernel and follow the instructions in the INSTALL file that came with the distribution. Read the instructions twice to compile your new kernel and the tools! After successful installation you are ready to step into the world of LVM. I would recommend that you reserve the space for one primary partition to use with LVM. The size of that partition can be as small as 100MB or as big as the whole disk - ideally you would use at least two. With two or more disks the more advanced examples are easier to follow ;-) and to do by yourself. But that is not needed to get used to LVM. It is even possible to use loop devices with LVM although that is no solution that will likely be used in a production setting. Lets begin with some easy but basic examples. 4.2 The first Physical Volume Since VGs are the containers of disk space we need to create at least one to start with LVM. The first step is to determine wich partition will be our first PV. The following output shows that there is one empty primary partition wich can be used: ~ # fdisk -l /dev/hda Disk /dev/hda: 240 heads, 63 sectors, 2343 cylinders Units = cylinders of 15120 * 512 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 203 1534648+ 7 HPFS/NTFS /dev/hda2 204 272 521608+ 83 Linux /dev/hda3 273 1356 8195008+ 7 HPFS/NTFS The partition /dev/hda4 is unused and contains 7461720 blocks which can be used for LVM. Our next step is to create the partition (using fdisk, cfdisk or any other competent partitioning tool) and set its type to ``0x8e''. After that the partition table will look like this: ~ # fdisk -l /dev/hda Disk /dev/hda: 240 heads, 63 sectors, 2343 cylinders Units = cylinders of 15120 * 512 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 203 1534648+ 7 HPFS/NTFS /dev/hda2 204 272 521608+ 83 Linux /dev/hda3 273 1356 8195008+ 7 HPFS/NTFS /dev/hda4 1357 2343 7461720 8e Unknown Now we need to make this partition a PV that can be recognized by the LVM tools: ~ # pvcreate /dev/hda4 pvcreate -- pysical volume ``/dev/hda4'' successfully created This can be checked with the command ~ # pvscan pvscan -- reading all physical volumes (this may take a while...) pvscan -- ACTIVE PV "/dev/hda4" of VG "rootvg" [7.11 GB / 7.11 GB free] pvscan -- total: 1 [7.12 GB] / in use: 1 [7.12 GB] / in no VG: 0 [0] and the command ~ # pvdisplay /dev/hda4 --- Physical volume --- PV Name /dev/hda4 VG Name rootvg PV Size 7.12 GB / NOT usable 6.84 MB [LVM: 126 KB] PV# 1 PV Status available Allocatable yes Cur LV 4 PE Size (KByte) 8192 Total PE 910 Free PE 238 Allocated PE 672 will show us what we created here. This is a partition that now can be used as part of a VG and thus we are ready to create the first Volume Group. 4.3 The first Volume Group The basis for the first Volume Group has been laid. Use the command ~ # vgcreate mynewvg /dev/hda4 to generate the Volume Group ``mynewvg'' and activate the VG with ~ # vgchange -ay mynewvg Now you can check the success with ~ # vgscan vgscan -- reading all physical volumes (this may take a while...) vgscan -- found active volume group "mynewvg" vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created vgscan -- WARNING: you may not have an actual backup of your volume group The result will be a VG with the size of 7.11GB, the default PE size set to 4MB. The following command will show these and other informations: ~ # vgdisplay mynewvg --- Volume group --- VG Name mynewvg VG Access read/write VG Status available/resizable VG # 0 MAX LV 256 Cur LV 0 Open LV 0 MAX LV Size 511.98 GB Max PV 256 Cur PV 1 Act PV 1 VG Size 7.11 GB PE Size 4 MB Total PE 1820 Alloc PE / Size 0 / 0.00 GB Free PE / Size 1820 / 7.11 GB What we have created now is an ampty Volume Group. This VG is extendable at any time with more PVs - up to a maximum of 256, if you like to issue the command pvcreate that often ;-). It is the stock of goods we told of in our previous example. Within this stock we can create now Logival Volumes. 4.4 The first LV Up to now we haven't had to do with actual data storage yet - but that is what it's all about. But now we are ready to set up our first Logical Volume which in turn will carry a filesystem that actually can be used. We will make it not that big for our first try but you will get the picture. Issue the command ~ # lvrceate -L100M -n mynewlv mynewvg lvcreate -- rounding up size to physical extent boundary "100 MB" lvcreate -- doing automatic backup of "mynewvg" lvcreate -- logical volume "/dev/mynewvg/mynewlv" successfully created