Kickstart
Contents
Policy
- Approval (by Head of Security Team) is required before creating any non-kickstart or non-company-approved standard Linux or Unix OS installation.
- Installation of company approved kickstart OS image must be used to create a standard Linux image.
- New installs must all be 64bit CentOS 6 (or newer) unless there is a special 3rd party requirement that it has to be 32bit. 32bit systems will also need an approval.
- Kickstart creates a minimal OS install, plus a few approved packages for troubleshooting purposes.
- Our standard is to allocate all available/remaining space to /home, with company applications and support software run from /home (see step 6 in the Running Kickstart section under Kickstart for additional details), to isolate disk usage of additional process/applications from OS processes & logs.
- Kickstart incorporates standardized partitioning/configuration/packages/security settings/etc.
- Systems must use a central DNS/NTP/MAIL/Proxy for the Data Center they are in.
- Systems must be attached to spacewalk and a central logging server.
Running Kickstart
- Preamble*
Instructions assume you have logged into one of the local kickstart servers (on example.net) and have sudo'd to the ks user, ex;
# ssh -p 222 sj-ks1 # sudo su - ks
1a. When building a new system, our standard is CentOS. Please try to use Enterprise 7 for everything, unless there is a specific requirement to use 6 (even when rebuilding older env's that were built under 5, per Dev Teams requirements, we've been building them under 6 instead).
When building a hadoop mgmt node, use the app image and specify these users;
hadoop hdfs mapred hbase zookeeper hue cloudera-scm hive
When building a hadoop slave node (data/jobtracker/mapreduce nodes), use the hadoop image for the hardware you're using, and specify these users (note: hadoop image now adds these users, add these users if not using the hadoop image);
hadoop hdfs mapred hbase
1b. Run the config build script (it is OK to ^C out of the process and run it again);
# mkks
2a. If you are building a VM, please configure at least the following minimum requirements (also required for physical systems, but they are normally higher than these values by default). Template:Defaults
2b. Connect a server to the build network. If this is a new VM with an new unformatted disk, you should get the PXE menu upon booting. Alternately, you can hit F12 during the BIOS screen to get the PXE menu. If your VM already has a formatted disk and you are having trouble PXE booting; hit F12 to PXE boot, hit ESC for boot menu, or delete and re-add your hard disk.
Note: use Firefox if kicking a Dell, IE has issues with the F12 button
3. choose the entry with the new hostname from the PXE menu
4. reboot
5. login to your new system, change the password (for the new/secure password settings, it was changed to; changeM3now!), and check for OS security updates;
# ssh -p 222 `newsystem` # passwd note: see the Updates section below about pushing updates back into kickstart for your next build # yum update
6. all avail space is allocated to /home, due to this, we would normally move other user/process dirs requiring space to /home (normally ice & mysql, do not move /var/www - this dir should not be used for any of our sites/functionality). Please do not change configs or home paths (passwd file), and please continue to refer to these dirs via their original location (/var/lib/ice, /var/lib/mysql, etc). There will be many many scripts/configs that would have to be changed, it's much easier to stay with their current path designations (as long as you have the link for the existing configs/scripts to continue working). Examples;
# cd /var/lib ; mv ice /home/ ; ln -s /home/ice # cd /var/lib ; mv mysql /home/ ; ln -s /home/mysql
7. cleanup; remove your new hostname from the PXE menu on kickstart, move your hostname.cfg file to the "old" dir (don't delete it), kickstart is now efi enabled, which has it's own/separate PXE menu (now there are 2 PXE menu files)
# mv ~/ks.cfg/hostname.cfg ~/ks.cfg/old/ On older 5.x systems (:wn to get to next file); # vi /tftpboot/pxelinux.cfg/default /tftpboot/efidefault On newer 6.x systems (:wn to get to next file); # vi /var/lib/tftpboot/pxelinux.cfg/default /var/lib/tftpboot/efidefault
8. maintenance; kickstart is highly customized for our company, and patching can break bug fixes. Please refer to /home/ks/scripts/* on any kickstart server as a reference. Of special note is the OSfixes script.
Suggestions
If you run into any issues or have suggestions, please email support, and/or add them to: Kickstart ToDo List
Layout
iso (OS) files are in ~/iso/ mkks configs are in ~/bin/ lists of profiles/configs avail are in ~/bin/mkks.lists list of iso's avail are in ~/bin/mkks.iso ks.cfg configs are in ~/ks.cfg/ software (non-OS) is in ~/software/ (java, nrpe, etc..) boot menu is at /tftpboot/pxelinux.cfg/default or /var/lib/tftpboot/pxelinux.cfg/default (depending on OS ver) boot (pxe) files are in /tftpboot/images/ or /var/lib/tftpboot/images/ (depending on OS ver) (the iso & pxe dir (images) tree's should be an identical directory structure)
Replicating kickstart files
tar up the master ks home dir, and use scripts/ksdiff to audit, when updating ks servers (should be a svn checkout/export);
cd ; tar cjvf /tmp/ks.tbz --exclude=.svn --exclude=iso --exclude=bin/mkks.conf --exclude=bin/mkks.iso --exclude=ks.cfg/comps --exclude=ks.cfg/old --exclude=ks.cfg/*.cfg --exclude=ks.cfg/*.prf --include=hostname.prf-sample* --exclude=ks.cfg/templates/old --exclude=stage --exclude=software/desktop *
Note: if creating a new kickstart server, a bin/mkks.iso will have to be created for the iso's that have been downloaded/copied to that system (see bin/mkks.iso-sample). Also, the tftp path may need to be modified in bin/mkks.conf (both examples are there for 6.x, or a pre-6.x env). The following packages are required; dhcp tftp-server xinetd syslinux (nfs is also, but is part of kernel / is already there). The ~/scripts/kson may be helpful, to startup services for the 1st time. Example configs are in ~/ks.cfg/etc/ (so they can be checked into subversion), and tftp boot files are in ~/ks.cfg/tftpboot/.
Updates
1. Updates can be pushed to the kickstart system and they'll get applied to a new system during build time. Normally the updates are not saved and are removed from the system after the update process. To save updates (so they can be pushed to kickstart), edit yum.conf and change keepcache to 1, before applying any of the updates;
# vi /etc/yum.conf
2. After updating your system, the update files can be pushed to kickstart. The files from the 'cachedir' defined in yum.conf (usually /var/cache/yum/updates/packages/ or /var/cache/yum/x86_64/6Server/rhel-x86_64-server-6/packages/ for RedHat 6.x) will need to be copied to the kickstart server, into the appropriate iso/`distro`/`ver`/updates dir (note, if you built a CentOS 5.2 system and have updated it to 5.3, put the updates under 5.2, NOT 5.3 - if you built a CentOS 5.5 system, the updates would go in iso/CentOS/5.5/updates/)
3. Make sure and turn off the keepcache setting and remove the leftover updates files;
# vi /etc/yum.conf # rm -f /var/cache/yum/updates/packages/* (or /var/cache/yum/x86_64/6Server/rhel-x86_64-server-6/packages/* for RedHat 6.x)
Files we touch
ISO Integration
1. periodically new Distro versions need to be added to the iso & tftpboot/images dir. Here is an example with a fictitious CentOS 9.7;
# mkdir -p ~/iso/CentOS/9.7/x86_64 # mkdir -p path-to-tftpboot-dir/images/CentOS/9.7/x86_64 # cd ~/iso/CentOS/9.7/x86_64 (download new iso(s) into this dir) # sudo mount -o loop CentOS-9.7-x86_64-bin-DVD.iso /mnt/cdrom # cp -p /mnt/cdrom/images/pxeboot/[org8io8public:iv]* path-to-tftpboot-dir/images/CentOS/9.7/x86_64/ (there is a new step for 6.x; mkdir images ; cp -p /mnt/cdrom/images/install.img images/ ; cp -p /mnt/cdrom/images/product.img images/) # sudo umount /mnt/cdrom (add new iso versions to /home/ks/bin/mkks.iso, convention is to have the highest/latest version only) # vi ~/bin/mkks.iso
New Kickstart Server Setup
1. Kickstart a new system with latest CentOS (currently 6.x) and add the ks user and the ksadmins group
2. Install needed packages
# yum install dhcp tftp-server xinetd syslinux
3. Grab lastest kickstart setup from subversion under ks user, OR, on sj-ks1, cd to and tar the /home/work/ks dir excluding the subversion tracking files (cd /home/work/ks ; tar cjvf /tmp/ks.tbz [!i]* --exclude=.svn)
4. As root, copy various configs;
# chmod o+r /home/ks # chown ks:ks /var/lib/tftpboot/ # cat /home/ks/ks.cfg/etc/exports >/etc/exports # cp -p /etc/dhcp/dhcpd.conf /etc/dhcp/dhcpd.conf-original # cat /home/ks/ks.cfg/etc/dhcpd.conf >/etc/dhcp/dhcpd.conf
5. setup secondary network interface on 192.168.0.1 per example NIC config;
# cat /home/ks/ks.cfg/etc/ifcfg-eth1 # vi /etc/sysconfig/network-scripts/ifcfg-eth1 # ifup eth1
6. startup services
/home/ks/scripts/kson
7. As ks user, copy various configs;
# rsync -a ~/ks.cfg/tftpboot/ /var/lib/tftpboot/
# mkdir -p /var/lib/tftpboot/images/{CentOS,RedHat,Oracle}
# mkdir -p ~/iso/{CentOS,RedHat,Oracle}
8. Create mkks.conf and mkks.iso files in bin from the sample files provided, mkks.iso has the list of iso files that were downloaded to the new system based on the steps in the ISO section (previous section above). The main build script (mkks) will also look for a mkks.lists-$DOMDEF (if it's set to xyz.com, it will look for a mkks.lists-xyz.com), which may be helpful if you're not using a company domain, but still syncing with a company kickstart.
Building Custom ISO/USB Flash Image
1. extract the DVD into a temporary work area
# mkdir /home/kickstart/iso ; cd /home/kickstart/iso # mount -o loop /home/kickstart/CentOS-7-x86_64-DVD-1511.iso /mnt/cdrom # rsync -a /mnt/cdrom/* . # cp -p /mnt/cdrom/.discinfo . IF there's a 2nd DVD: # mount -o loop /home/kickstart/CentOS-6-x86_64-DVD2.iso /mnt/cdrom # rsync -a /mnt/cdrom/Packages/* Packages/
2. modify isolinux/isolinux.cfg file 2a. set timeout, change the following (timeout 0 means to never timeout, so don't set it to 0 unless you want to sit at the menu, and not jump right into the automated installation);
FROM: timeout 600 TO: timeout 1
2b. change 'linux' menu section;
NOTE: label must match the original distro iso's label
CHECK/ADD (under "menu label .." line), if not there already: menu default CHANGE, remove the "quiet" argument on the append line, ex: append initrd=initrd.img inst.stage2=hd:LABEL=CentOSx207x20x86_64 quiet TO, and add a kickstart line: append initrd=initrd.img inst.stage2=hd:LABEL=CentOSx207x20x86_64 inst.ks=hd:LABEL=CentOSx207x20x86_64:/ks.cfg/ks-c7a.cfg
2c. remove the following line from all other menu sections (like the DVD "check" menu);
menu default
3. copy the needed files from a kickstart system (the software tarball will be quite large); Note: in your temporary work area for building your iso or USB image, you might want to remove 'software' files for other releases, i.e. if you are building a CentOS 7 image, remove all the el5 & el6 files, ex; rm -f software/*/*.el[56].*
IN /home/ks on a kickstart system; # tar cjvf /tmp/ks-min.tbz ks.cfg/ks-c7a.cfg scripts # tar cjvf /tmp/ks-sw.tbz software IN /home/kickstart/iso on your build system; # tar xjf /home/kickstart/ks-min.tbz # tar xjf /home/kickstart/ks-sw.tbz REMOVE the following; # rm -rf software/certs.prd (assuming you're not installing any of these) # rm -rf software/jboss # rm -rf software/oracle # rm -rf software/hornetq # rm -rf software/ice # rm -rf software/vmware IF not needed (if not doing a Desktop install/etc); # rm -rf software/desktop SETUP users & groups files (remove unneeded users, add needed users); # vi scripts/mkusers/users scripts/mkusers/groups REMOVE rest of company user info; # rm -f scripts/mkusers/users[.-]* # rm -f scripts/mkusers/ufx-* # rm -f scripts/mkusers/groups-master
4. build the bootable media 4a. build a DVD .iso file. make sure the mkiso software is installed, and run the mkiso command;
# yum install genisoimage # mkisofs -o /home/iso/CentOS-7.2-x86_64-kick-20160114a.iso -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -V 'CentOS 7 x86_64' -boot-load-size 4 -boot-info-table -R -J -v -T iso/ OR
4b. build a bootable USB Flash drive. plug a USB Flash into your system, make it linux bootable (will be ext2)
# fdisk /dev/sdb # mkfs /dev/sdb1 NOTE: label must match the original distro iso's label # e2label /dev/sdb1 "CentOS 7 x86_64" # yum install syslinux syslinux-extlinux # cat /usr/share/syslinux/mbr.bin >/dev/sdb # mount /dev/sdb1 /mnt/cdrom/ # rsync -a iso/ /mnt/cdrom/ # mv /mnt/cdrom/isolinux /mnt/cdrom/syslinux # mv /mnt/cdrom/syslinux/ # cd /mnt/cdrom/syslinux/isolinux.cfg /mnt/cdrom/syslinux/syslinux.cfg # extlinux -i .
Optional: Minimizing packages (.rpm's) To save room on the DVD disk or USB Flash drive (and to make it easier to copy/replicate), you might want to minimize the repository of available packages (after the install it pulls any new packages from the internet). The easiest way to determine the list of packages needed, is to do an install based on your current ks.cfg. Once you have an installed system, you can just save the 'rpm -qa' output and remove all files except the ones needed for your install (this saved us over 3.5G on our first DVD).
# cd Packages # for OLDFILE in * ; do FILE=$(echo "$OLDFILE" | sed 's/.rpm$//') ; if ! grep "^$FILE$" /home/kickstart/rpm-qa_c7a.txt >/dev/null ; then echo "deleting $OLDFILE.." ; rm -f $OLDFILE ; fi ; done # cd .. # cp repodata/*-comps.xml.gz ../comps.xml.gz # gunzip ../comps.xml.gz # yum install createrepo # createrepo -g ../comps.xml . # rm -f repodata/*-comps.xml*
Setting grub password
Enterprise 7 1. The following command will ask for a password and change grub config files (creates /boot/grub2/user.cfg)
# grub2-setpassword
Ent 4/5/6 1. create password hash. The following will ask for a password and display a sha512 hash of the password
# grub-crypt
2. edit grub config and add/change the password
# vi /boot/grub/grub.conf ADD/edit the following line (should be after the timeout line) password --encrypted $6$XrbtG7T7KxjKCYEz$uo4QGG9d5kEJkvVv7JBg3jB.kphXGpanpdDj7MGqsHHcdZpCOUP2dJjFI1db7RdcD.CzesOpmEEIjdqwAOEY2/
Reference
Disks >2Tb
when using kickstart on systems that have drives greater than 2.2Tb (tested on ent 6, probably not needed on ent 7), add the following to the ks.cfg profile that was generated for the system being kicked (if modifying partitions on these systems, use gdisk, don't use fdisk)
%pre parted -s /dev/sda mklabel gpt %end
Legacy eth* NIC Names
to force NIC naming to legacy eth* convention, use the following kernel params
BEFORE ENT 7 biosdevname=0 ENT 7 net.ifnames=0
