UPS HOWTO Eric Steven Raymond [http://www.catb.org/~esr/] Thyrsus Enterprises Revision History Revision 1.1 2004-02-21 Revised by: esr Add a little deployment advice, and stuff on other technologies. Revision 1.0 2003-10-07 Revised by: esr Initial release. An Uninterruptible Power Supply (UPS) is an important thing to have if you live in an area where power outages are at all common, especially if you run a mail/DNS/Web server that must be up 24/7. This HOWTO will teach you things you need to know to select a UPS intelligently and make it work with your open-source operating system. ----------------------------------------------------------------------------- Table of Contents 1. Introduction 1.1. Why this document? 1.2. New versions of this document 1.3. License and Copyright 2. An Overview of Power Protection 2.1. Surge suppressors 2.2. Line Conditioners 2.3. Uninterruptible Power Supplies 3. UPS Basics 3.1. How To Select A UPS 3.2. Deploying your UPS and other devices: the total picture 3.3. Software Assistance 3.4. Preparing Your System For Auto-Reboot 4. Maintaining Your UPS 4.1. Extending battery life 4.2. Recalibrating Your UPS 4.3. Replacing Your Batteries 4.4. Buying Batteries 5. Acknowledgements and Related Resources 1. Introduction 1.1. Why this document? An Uninterruptible Power Supply (UPS) is an important thing to have if you live in an area where power outages are at all common, especially if you run a mail/DNS/Web server that must be up 24/7. The aging power grid in the U.S. has made this a more urgent issue than it used to be even for American hackers, but everyone is vulnerable to outages caused by storms and other natural phenoena. This document covers both the software and hardware aspects of protecting yourself. The advice in this document is aimed primarily at small installations ?? one computer and one UPS. Thus we'll focus on consumer-grade UPes, especially those designed for home and small-business use. If you are a data center administrator running a big server farm, there is a whole different (and much more expensive) range of technologies we'll do no more than hint at here. ----------------------------------------------------------------------------- 1.2. New versions of this document You can also view the latest version of this HOWTO on the World Wide Web via the URL [http://www.tldp.org/HOWTO/UPS-HOWTO.html] http://www.tldp.org/HOWTO /UPS-HOWTO.html. ----------------------------------------------------------------------------- 1.3. License and Copyright Copyright (c) 2003, Eric S. Raymond. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is located at [http://www.gnu.org/copyleft/fdl.html] www.gnu.org/ copyleft/fdl.html. Feel free to mail any questions or comments about this HOWTO to Eric S. Raymond, . But please don't ask me to troubleshoot your general UPS problems; if you do, I'll just ignore you. ----------------------------------------------------------------------------- 2. An Overview of Power Protection There are three levels of power protection available to the home computer user. The levels are: 1. Surge Suppressor 2. Line Conditioners 3. Uninterruptible Power Supplies While this HOWTO mainly focuses on UPSs, we'll start with some basics about the other two kinds of power filtering to help you understand where UPSes fit in. This is useful even though plummeting UPS prices have made the low-end alternatives less interesting than they used to be. ----------------------------------------------------------------------------- 2.1. Surge suppressors These are basically a fancy fuse between the source and your hardware; they clamp down spikes, but can't fill in a low voltage level or dropout. This is a bare minimum level of protection that any piece of expensive electronics should have. Note that this applies to more than just AC power; surge suppressors are available for (and should be used on) phone lines, and RS-232 and parallel connections (for use on long lines; generally not needed if the devices are colocated with the computer and all devices are protected from outside sources). Note also that all devices connected to your computer need to be protected; if you put a surge suppressor on your computer but not your printer, then a zap on the printer may take out the computer, too. An important fact about surge suppressors is that they need to be replaced if they absorb a large surge. Besides fuses, most suppressors rely on on components called Metal-Oxide Varistors (or MOVs) for spike suppression, which degrade when they take a voltage hit. The problem with cheap suppressors is that they don't tell you when the MOV is cooked, so you can end up with no spike protection and a false sense of security. Better ones have an indicator. You can buy surge suppressors at any Radio Shack; for better prices, go mail-order through Computer Shopper or some similar magazine. All of these are low-cost devices ($10-50). ----------------------------------------------------------------------------- 2.2. Line Conditioners These devices filter noise out of AC lines. Noise can degrade your power supply and cause it to fail prematurely. They also protect against short voltage dropouts and include surge suppression. The Tripp-Lite 1200 I used to have was typical of the better class of line conditioners; a box with a good big soft-iron transformer and a couple of moby capacitors in it and no conductive path between the in and out sides. With one of these, you can laugh at brownouts and electrical storms. A fringe benefit is that if you accidentally pull your plug out of the wall you may find you actually have time to re-connect it before the machine notices (I did this once). But a true UPS is better. Netter Trey McLendon has good things to say about Zero Surge conditioners. He says: "Our systems at work [...] have been protected for 2.5 years now through many a violent storm...one strike knocked [out] the MOV-type suppressors on a Mac dealer's training setup across the street from us. The Zero Surge just sort of buzzed when the surge came in, with no interruption whatsoever. The basic principle is this: ZS units slow down the surge with a network of passive elements and then sends it back out the neutral line, which is tied to ground outside at the box by code. MOV units shunt the surge to ground at the computer, where it leaps across serial ports, network connections, etc. doing its deadly work." Price vary widely, from $40-400, depending on the power rating and capabilities of the device. Mail-order from a reputable supply house is your best bet. Line conditioners typically don't need to be replaced after a surge; check to see if yours includes MOVs. ----------------------------------------------------------------------------- 2.3. Uninterruptible Power Supplies The remainder of this document will focus on UPes. A UPS does three things for you. First, it filters the power your machine sees, smoothing out spikes and voltage fluctuations that can stress or even damage your electronics. Secondly, it provides a certain amount of dwell time in the event your power goes out entirely ?? this can often get you through brownouts and short blackouts. Third, when the UPS is about to run out of power it can arrange a graceful shutdown of your computer so that no unpleasant things happen to your disk filesystems. While the risks of unexpected shutdown are much lessened in these days of journalling filesystems like Linux's EXT3 or JFS from what they once were, ensuring a clean shutdown is still a valuable contribution to any system administrator's peace of mind. ----------------------------------------------------------------------------- 3. UPS Basics 3.1. How To Select A UPS UPSes are nowadays very inexpensive. In the U.S. in 2003, quite capable ones are available for less than $150, and prices are heading down. In fact prices are so low now that we're not going to walk you through the elaborate optimization step that would have been important even two or three years ago, of estiming the watt dissipation of your computer and matching it to a UPS rating. Instead we'll explain why this would be a waste of effort and how to buy in a simpler and more effective way. UPSes are rated by the watts a full battery can put out before it drains. However, they are marketed using a VA (voltage-amps) figure; often, consumer-grade UPSes don't even specify a wattage on the box where you can see it. This is because the VA figure is larger and looks sexier. As a rule of thumb. assume the wattage is half of the VA rating; for an explanation of the complexities involved (if you care) see the white paper Understanding Power Factor, Crest Factor, and Surge Factor on the APC website. But even if you know the watt rating of the UPS, it is the ratio of that figure with the wattage dissipation of your computer that controls the dwell time. Your dissipation is hard to predict; it can even be effected by things like the size of monitor you use (big ones can be quite power-hungry). Manufacturers try to get around this technical thicket by putting an expected dwell time on the box. But they exaggerate and even lie about their dwell times a lot (this is called "marketing"). What they'll do is quote you the dwell time you would get driving a bare minimum system with the disk drives shut off and a tiny monitor, in much the same way laptop manufacturers lie about their battery dwell times. The more honest UPS manufacturers give you a little table showing expected dwell times for different system configurations ("desktop", "tower", etc.). As a rule of thumb, assume you will get about 50% of the dwell time listed on the box for your configuration type. My advice is to forget the numbers game. Just go online or to your local computer store and buy one of the higher-end consumer or home-office models from APC, Best, Tripp-Lite, Belkin, or some other reputable manufacturer. Go ahead and grab the model with the longest dwell time, highest watt rating, or biggest VA number you can find; the premium for it is not likely to be more than $75 over the bargain-basement model. I guarantee you will feel very good about your decision not to pinch pennies come your first extended power outage. Perhaps a more compelling reason it is better to over-buy capacity rather than ending up with a UPS that is too weak for your power drain is that overstrained UPSes can fail in ugly ways, including catching fire and exploding. Be sure you get a line interactive UPS rather than the older standby or SPS type. The older technology doesn't actually filter your power through the battery, so you're not assured of good voltage conditioning. The main advantage of an SPS (low cost) has been eroded now that line-interactive UPSes are so inexpensive. There are other UPS types, but they are either obsolescent or targeted at large data-center installations. For a detailed discussion of the different UPS types, see The different types of UPS systems , a white paper on the APC site. Another important consideration is how your UPS will communicate with your computer. Do not buy a serial line UPS (one that communicates via an RS-232C cable). These are passing out of use in favor of UPS designs that use USB or Ethernet, for the very excellent reason that RS-232C interfaces are flaky, difficult to configure, and difficult to debug. Ethernet is overkill for this application; UPSes simply don't need that kind of bandwidth. We recommend sticking with USB, which is well-matched in price/performance to this job and relatively easy to troubleshoot. Until recently there was an important distinction between smart and dumb UPSes. Dumb UPSes did voltage-level signaling through individual pins; smart ones used the link as a primitive character channel and could pass more status information over it. But if you avoid RS232C UPSes you will never see a dumb one; indeed, it is likely that by the time you read this no dumb UPses will be in production any longer. Personally, I like APC UPSes (I have no connection with the company). But this is not the kind of widget for which manufacturer makes a whole lot of difference as long as you stick with one of the reputable brands. ----------------------------------------------------------------------------- 3.2. Deploying your UPS and other devices: the total picture Our recommendation for a production Unix environment is a configuration like the following: 1. An UPS for the computer system. 2. Surge suppression on all phone lines, and also on serial/parallel lines that leave the room. 3. Line conditioners on any devices not connected to the UPS. If you do take a power hit, it's cheaper to replace a $50 line conditioner than a $1500 laser printer. If this is too expensive for you, then downgrade the UPS to a line conditioner like the TrippLite. But don't go without at least that. Running unprotected is false economy, because you will lose equipment to electrical storms ?? and, Murphy's Law being what it is, you will always get hit at the worst possible time. One thing to note is that you typically shouldn't put a laser printer on a UPS ?? toner heaters draw enough current to overload a UPS and cause a shutdown within seconds. The other thing is that you can't even put the laser printer on the same circuit with a UPS ?? the heater kicks on every 20-30 seconds, and most UPSs will see the current draw as a brownout. So buy a separate line conditioner for the laser printer. A UPS should be wired directly to (or plugged directly into) the AC supply (i.e. a surge suppressor is neither required nor suggested between the wall and the UPS). In addition, a surge suppressor between the UPS and the equipment connected to it is redundant. ----------------------------------------------------------------------------- 3.3. Software Assistance Your UPS communicates with your computer so it can gracefully shut the computer down when an outage has lasted too long for the battery to cope. In order for graceful shutdown to actually happen, your computer needs to have a background process ?? a daemon, in Unix terms ?? watching whatever messages come over the UPS cable for the one that says terminate. Then it needs to tell the operating system to shut down. Your UPS probably comes with a CD full of such software. Throw it away, as (a) most of it will be useless bits written for Windows systems, and (b) in the unlikely event you get Linux software it will almost certainly be stale binaries for a version you don't run. Back in the days of dumb serial-line UPses, there used to be about half a dozen different open-source UPS monitor daemons: apcd, dumbupsd, genpowerd. powerd, smupsd, usvd and more. These were fairly stupid programs for a simple job. Many required you to hand-wire a custom RS232C cable to get around various evil things that UPS manufacturers did to their ports in order to lock in customers. Those days are gone. USB UPSes get rid of the cable-hacking and hardware klugery, but require a bit more smarts from a monitor daemon. Accordingly the field has narrowed considerably. There appear to be only two such projects left standing. The Network UPS Tools project is a generic UPS monitor daemon that aims to communicate intelligently with all current UPS designs. [http://www2.apcupsd.com/] apcupsd is a daemon specifically designed for communicating with UPSes made by APC, the American Power Corporation. Both are solid, well-run projects. Their development groups are mutually friendly, and there has been occasional talk of a merger. Awkwardly, as of October 2003 the apcupsd project is the more featureful of the two, with, among other things, better USB support and better documentation ?? but the NUT tools have a cleaner architecture, more developers, and acceptance in Red Hat and other major distributions. My advice is simple; run apcupsd if you buy an APC UPS, and the NUT tools if you buy anything else. RPMs and Debian packages (which will modify your system's boot sequence appropriately as well as installing the daemon binaries) are available for both, so installation should be easy either way. ----------------------------------------------------------------------------- 3.4. Preparing Your System For Auto-Reboot If you are using your UPS to try to keep a DNS/Web/mailserver up 24/7, you will want to make sure the machine can be configured to boot automatically when it is powered up. This is not the normal behavior of most computers as shipped from the factory. Normally after the power is cut and restored, you must explicitly press a button for the power to actually be turned on. You can test your computer by powering it down; shutting off the power (pull the plug); then plugging the cord back in. If your computer immediately starts up, good. There is nothing more to do. If your computer does not start up, manually turn on the power (by pressing the power on button) and enter your computer's SETUP program (often by pressing DEL during the power up sequence; sometimes by pressing F10). You must then find and change the appropriate configuration parameter to permit instant power on. Normally, this is located under the BOOT menu item, and will be called something such as Restore on AC/Power Loss or Full-On. The exact words will vary according to the ROM BIOS provider. Generally you will have three options: Last State, Power On, and Power Off. Some BIOSes do not support such an option. This is idiotically bad design, but it does happen. If so, your only practical remedy is to get a new motherboard. ----------------------------------------------------------------------------- 4. Maintaining Your UPS Your UPS has a battery inside it. Usually it is a lead-acid type (those are the least expensive for the manufacturer), but both lithium and gel-cel batteries are sometimes used. The battery is by far the most vulnerable and failure-prone part of your UPS. If you have your UPS long enough, you will probably have battery problems. Once every six months to a year or so you should recalibrate your UPS's battery sensor, and once every several years you will have to replace the batteries. ----------------------------------------------------------------------------- 4.1. Extending battery life To extend your battery life, (a) avoid deep discharges, and (b) don't expose them to extremes of heat, cold, or humidity. Unfortunately there is not much you can do to avoid deep-discharging your UPS other than living in an area where power outages are few and short. ----------------------------------------------------------------------------- 4.2. Recalibrating Your UPS Your UPS's dwell-time calibration will lose accuracy over the life of the battery. The usual symptom of this problem is that the UPS overestimates the dwell time it has remaining during outages, but occasionally it can also lead to an actual bad-battery condition going undetected and very odd symptoms as a result. UPSes have a recalibration procedure built into their firmware. It generally involves deep-discharching and recharging the battery while the UPS is in a special test mode. Your recipe for triggering such a recalibration will vary according to your UPS software. You always need to do this when you install new batteries (see below). It is a good idea to do it once every six to twelve months as routine maintenance, but no more often than that; as we noted previously, deep discharges shorten your battery life. ----------------------------------------------------------------------------- 4.3. Replacing Your Batteries All modern UPSes have a low-battery alarm and run a periodic self-test; they will alert you when replacement is needed. Usually they both flash an indicator and make an alarm sound. If you have a monitoring daemon set up, they will alert it and you will probably get warning mail. If you ignore the alarm it will time out, but be repeated at intervals. You will occasionally get a false alarm. It's a good idea, if you get an alarm, to explicitly trigger a UPS self-test the next day and see if the alarm goes away (the procedure for doing this varies depending on your UPS software). If the alarm is persistent, you need to replace the batteries. It has been reported that bad batteries can also produce symptoms that mimic inverter failures or wonky control electronics. Even if your UPS is displaying epileptic symptoms like repeating alarms and flashing panel lights, a bad battery is the first thing to suspect. UPS manufacturers would of course prefer that you replace your entire UPS when the batteries die, since they make more money that way. But in fact there is nothing unique or magic about UPS batteries. They are standard types also used for other applications such as powering marine electronics, with standard connectors. You can buy them from sources other than the UPS manufacturer, and sometimes replace them with equivalents that are better and less expensive. It's best to wait until the low battery alarm before ordering a replacement; keeping batteries on the shelf reduces their life unless you keep them fully charged. Do not throw old batteries in your regular trash! They contain toxic metals and acids. Be kind to your environment and hand them to a qualified party for recycling. Most battery dealers will cheerfully do this for you. If not, your local garbage company or waste-disposal authority can explain to you how and where to turn them in safely. Below, you will find some suggestions for buying replacement batteries. One important note of caution: at least one user purchased one of the aftermarket batteries noted below and found out that they would not fit into his unit. This required cutting and soldering and other very undesirable things, so be extremely careful in measuring your batteries ?? including every millimeter of the terminal connections, which can cause problems. Although you can do a hot swap of your batteries while the computer is running, it may not be very satisfactory, because the unit will not know that the batteries have been swapped and your monitor daemon will continue to show a low-battery indication. To correct this situation, you must do a discharge and recharge of the battery. At that point the battery should be calibrated better. It may take several discharges and recharges of new batteries before they reach full capacity and the dwell-time calibration is accurate. If your UPS contains two or more battery units and your monitoring software reports separate voltage levels for them, one way to tell is to watch the divergence in voltage levels. As the cells reach nominal full capacity, their voltages should converge. ----------------------------------------------------------------------------- 4.4. Buying Batteries APC makes "Replacement Battery Units" for each of the SmartUPS models, but they sell them directly only in the U.S. Your local Yamaha SeaDoo shop (if you have one) carries 35 ampere-hour deep cycle marine batteries that are direct replacements for the kind APC uses in many of its models. These are gel-cel and will double the runtime and/or cut your recharge time in half. Here are some West Coast sources: Jet Works 1587 Monrovia Ave. Newport Beach CA 9266? Tel: +1 714 548-5259 J-W Batteries, Inc. Tel: +1 714 548-4017 WPS 49-1200 GEL-CELL KB-35 BATTERY The company I've heard most strongly recommended (by Carl Erhorn, a core developer on the apcupsd project) is called Battery Wholesale Distributors of Georgetown, Texas. If you have questions, you can reach them by phone at (800) 365-8444, 9:00AM to 5:00PM (their local time), Monday through Friday. Carl reports having gotten email from them on the weekends, although the office is not open then. The web site, with current pricing, is [http://www.batterywholesale.com] www.batterywholesale.com. They will ship outside of the US, they take all the usual credit cards, and they accept orders by phone or Web. Carl reports that BWD has found manufacturers who make batteries in the standard case sizes, but have additional capacity over original UPS batteries. Often, the difference is as much as 15% or so, and this can result in additional runtime. It's a nice upgrade for a minor increase in price. BWD is also 'green-aware', in that they encourage you to recycle your old batteries, and will accept the old batteries back from you if you cannot find a local place that recycles them. You pay the shipping but other than that, there is no charge. Carl says "I've been very pleased with their products, service, and pricing. I hope you find them as helpful to you as I do. I've been dealing with them since about 1994, and have never been disappointed. The owner of the place also is very good on technical issues, so if you have questions on their products, he can get as technical as you need to go." ----------------------------------------------------------------------------- 5. Acknowledgements and Related Resources Substantial portions of this document, notably the bits on maintaining your UPS, were originally part of the apcupsd documentation. The project maintainers have graciously permitted me to re-use them here. Other parts were part of my Unix Hardware Buyer HOWTO. There was a previous UPS HOWTO by Harvey J. Stein, last updated in 1997. It was so out of date that I ended up using none of it. There is [http://www.jetcafe.org/~npc/doc/ups-faq.html] UPS FAQ which is slightly dated but still contains some good advice.