IPMI Installation (Wheezy)

This is a step-by-step process for setting things up over IPMI. It is not terribly detailed, just some 'oops' or 'what is going wrong!?' situations.

Initial Steps

1. If you aren't connecting using a VPN (and possibly even if you are), most IPMI software has ssl support, so access through https.
2. Create a new username/password pair.
3. Some hosts get anal about what accounts you can nuke for IPMI settings. Make things as tight as you are allowed. Reasonably though, they are the ones with physical access. I really do try to get rid of the default login names, however.
4. SuperMicro iKVM seems to 'bind' to a single computer... You may need to reset it if you get 'connection failed'.
5. Grab the NetInst CD Image and mount it in iKVM.
6. You'll probably want to set the boot orders so the non-UEFI IPMI CDROM comes first.

Over-IPMI OS Installation

1. Advanced Options -> Expert install
2. Language settings -> Add single byte version of locale, set single byte as default.
3. Keyboard settings, 'CDROM' detect/mount - straightforward
4. Components to load: choose-mirror and parted-udeb
5. Network detect/configure - don't autoconfigure. If using CIDR appending be sure to base from the second IP given to you. Use Google's nameservers if others aren't provided (8.8.8.8 8.8.4.4)
   1. If you're having difficulty, be sure you're using the right interface. Not every host plugs in eth0.
6. Hostname: Most people choose a scheme of some sort, at least for lower numbers of machines.
7. The Steadfast mirror is quite handy if your machine is actually at Steadfast or a reseller (e.g. HostVenom).
8. Enable shadow passwords, allow root login, give good password (Mine are alphanumeric, mixed case, 12 characters, with symbols.
9. There's a pattern to my admin usernames, but I prefer to keep them secret. Generally use a temp password, set it to something ridiculous later as only logins will be via ssh keys
10. Setup clock, configure. Should probably consider standardizing on UTC time.
11. Detect disks, partition manually. Use gpt for new disks (GUID partition table), though may want to stick with msdos for ssds to help with overprovisioning (and some ssds don't support gpt at all), if you can't make use of the manufacturer's provisioning tool.

Partitioning

When setting up partitions, your first priority is, if you have a lot of disk I/O, to divide it between your drives accordingly. If you are going to have a database-heavy server, this will likely be, as of MySQL 5.0-5.5:

• Your InnoDB Databases (with file_per_table on) - SSD is best for this
• Your InnoDB double buffer log (the 'ibdata1' file with file_per_table off) - SSD still best, if suitably modern, but you can use a platter drive. It's possible to split this further in MySQL 5.6 and some of its forks, apparently, but I have not properly tested this division yet.
• Everything else, including other MySQL components - binlogs, ib_logfilex files, standard MySQL logs, etc.

To demonstrate just how even InnoDB and its doublewrite logs are, here is my iostat output this morning:

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               1.48        19.97        14.43  458827269  331716822
sdb              12.56        11.65       189.09  267668301 4345400908
sdd               4.56         1.15       402.10   26347310 9240368712
sdc              31.84       105.00       405.25 2413001385 9312856280

This probably seems obvious to most, but I have seen people argue against it, for reasons I cannot fathom.

If you only have two physical volumes, dividing up 'everything else' can be difficult, but the majority is likely going to be between binlogs and backups. In this case, backups are getting sent straight to the slave server - but if you are doing that sort of thing, you can probably afford four drives on your master server.

Your next priority is to make sure that filling up one partition won't bring your server down. I have had runaway files fill up /home on me, and I have heard many first-hand accounts of /var filling up - but our sites were still live, even if something was obviously wrong with them. People more experienced with LVM than I am will suggest that route, and for sda above, I would certainly consider it. sdb is a 4gb swap partition and /var.

swap I use 4GB for swap, and have for the better part of a decade now. Regardless of configuration, or total RAM, for my servers, about a half a gigabyte can be swapped out with no meaningful impact. Past that point, it starts eating into cache, and gets progressively more painful, and users really start complaining when it gets between 2GB and 3 GB.

Your mileage will of course vary, but don't be fooled by 'swap is useless' evangelism. Half a gig of free RAM is still free, and can give you significant breathing room on smaller servers.

noatime goes on all the things, except swap.

nodev, nosuid goes on everything not /boot or / (or swap)

noexec should be treated with caution. The installer likes to execute things out of /var, apt likes to execute things out of /tmp. I generally just place it on my storage /srv mounts.

I need to do some more testing with discard, but I am suspecting that yes, it is a performance drag rather than an improvement in modern SSDs.

Over-IPMI OS Install - Finishing

1. Installing base system - non-versioned kernel (for upgrades), targeted initrd (not going to be changing hardware much, if ever).
2. I prefer to turn popularity contest on, personally.
3. When getting to the abominable Tasksel/Software selection screen, deselect everything except for SSH server and Standard system utilities.
4. Install grub to MBR
5. Reboot, login as root
6. /etc/init.d/ssh stop
   1. We're listening on the standard port to start. No need to let it get hammered.
7. vi /etc/network/interfaces
   1. Setup remaining interfaces - or at least one for your desired sshd binding.
   2. Restart networking, ifup eth# if necessary.
8. vi /etc/ssh/sshd_config
9. Set new port and specific IP to bind to.
   1. Not really doing this to hide (though our IPTables will help with that). If you ever run a major server, though, you are going to get a lot of attempts at port 22, and a fair amount on various common server ports, and it clutters your logs. Just set it somewhere out of your way. My preference is to pick a single port number for all of my servers and stick with that.
10. PermitRootLogin no
    1. This is just common sense.
    2. If you're going to be copy/pasting a key from your local machine, leaving PasswordAuthentication on for a minute is not going to kill you.
11. X11Forwarding no
    1. This system will never use X, or any graphical manager.
12. UsePAM no
    1. ssh for life. Or something.
13. /etc/init.d/ssh start
14. Sign in as your admin user over SSH.
15. If works, can exit out of IPMI

Debian Wheezy

Initial: Designing a Server - IPMI Installation

---

Core: Network - OpenSSH - Packages - sysctl.conf - Iptables - Security - User Management

---

Stack: Unbound - MySQL - Nginx - PHP (5.4)

---

Mail

Mail Setup - Mail Tables - OpenDKIM - Mail and DNS - Spamassassin - ClamAV - Postfix - Dovecot - Reputation Management

---

Logs and Security: Logging - Logcheck - Backups - AppArmor