Ups power supply repair. What did I do?

Introduction: Make Your Computer UPS Last for Hours Instead of Minutes

For what would appear common sense to me, but perhaps not to all, I have all of my computers on UPS battery backups. After getting frustrated when the power flickered one day, I immediately went out and purchased a UPS. Well, shortly after, the power went out for longer than the battery could keep my computer afloat. I needed a better solution!

I wanted my UPS to be able to last for at least 60 minutes in a power outage. I needed more power! My solution: Car batteries.

Materials: UPS that is rated at least double what you plan to draw (see step 8 to understand why). Wire (12 awg or larger; two different colors) Solder Heat shrink tubing Car battery with terminals on the top Adapters to go from the car battery terminals to threaded rod. Wing nuts the same size as this threaded rod Wire crimp terminals that will fit over the threaded rod. Plastic case for your car battery Inline Fuse holder (radio shack) 30 amp fuse for holder (any auto store)

power, supply, repair

Tools: Screwdrivers Wire cutters Wire strippers Soldering iron Scissors (optional) heat gun or alternative Drill Drill bits

Step 1: Evaluate Your Needs

I was trying to power two computers (desktop and file server), and two flat panel monitors. My total power consumption was roughly 500 watts peak. (yikes!) Currently I was running on two 300 watt UPS’s (NOTE: VA is not equal to WATTS. Find the WATT rating) with one computer and one monitor on each. Even though the two monitors were hooked up to the same computer, I needed to distribute my power load more evenly to get longer battery life out of my petty UPS’s.

CAUTION:I discovered the hard way after nearly starting a fire and destroying a UPS that you need one that is rated at at least twice the wattage you are consuming. They can’t handle being run for longer than a few minutes at this rating, but the batteries die before it’s a problem normally.

So I now knew I needed 500 watts, and I wanted 60 minutes of power. that means:P / V = I500 watts / 120 volts = 4.16 ampere hours (at 120 volts)

UPS batteries are usually 12 volts, but some are wired with two batteries in series. Check yours out first to make sure you won’t need two car batteries.

So, assuming 12 volts, that means that, after adjusting for the voltage differences, I need a battery with at least 41.6 ampere hours. (yeah, I know there’s inefficiencies in the UPS, but lets keep math easy)

Step 2: Remove Battery From UPS

Unplug the UPS from the wall, and unplug all devices from it.Remove any screws you fine, and open up the case.If you are as lucky as I was, the battery will have terminals that you can slide off. If not, just cut the wires as close to the battery as you can.Once you have removed the battery, you will find something like you see in the picture

NOTE: Pay attention to polarity on the battery, and which wire went to when polarity.

The background

I have a UPS (uninterruptable power supply) for my home server, so that:

  • if the power goes out for a quick moment, my home server keeps going without skipping a beat.
  • if the power goes out for a long time, my home server gracefully turns itself off before the battery runs out.

I use my server for home automation and file storage (family photos, media, and backups), and the hard-stop of losing power cause irreperable data loss, which is bad.

Having a UPS solves all of those problems. I use a CyberPower Value600EI-AU UPS that I purchased for 80 AUD about 5 years ago. https://www.umart.com.au/product/cyberpower-value600ei-value-soho-600va-360w-simulated-sine-wave-ups-33214

The fix

I read somewhere that this issue can sometimes be caused by the battery dying. The UPS is about 5 years old after all. I figured I’d try to replace the battery before buying a whole new UPS.

I opened up the UPS by preying off the face panel and removing the 2 screws from the bottom of the case. Once the case was open, I could see the battery at the front. I also noticed that there was some buildup of stuff on the. negative terminal. Could it be battery acid or corrosion?

The battery that the UPS came with is a LEOCH 12v 7.2Ah SLA (sealed lead-acid) battery. Testing with my voltmeter showed the abttery only had around 7v, that’s not great for a 12v battery.

Luckily, the battery form-factor and dimensions are pretty common. In fact, they are the same battery used in Australian NBN battery backup boxes. I purchased a new battery of the same size and specs from Jaycar, a 12v 7.2Ah SLA for 34.95 AUD. https://www.jaycar.com.au/12v-7-2ah-sla-battery-nbn-back-up-battery/p/SB2486

Replacing the battery was easy enough. I removed the 2 leads connecting the old battery, and it wiggled right out.

Before plugging the new battery in, I wanted to clean up the stuff from the. negative lead and get some of that buildup off. I used a brass metal brush and my fingernails to get the buildup off. It’s much cleaner now, and should conduct electricity better.

I installed the new battery into the case, connected the 2 leads to the new battery, and it started up like a champ! It’s working as good as new so far. So a dead battery turned out to be the cause, and a 35 replacement part beat a 80 replacement for the whole thing.

Here’s a video of the UPS after replacing the battery.

Long part numbers and a long history

I decided to try pulling the sticker off of the microcontroller. What did I have to lose? Not an awful lot, the thing was already dead. Guess what I found underneath?

Image credit: eBay. liaoxiyuan

The part in question was actually a NXP P89CV51RC2FA. a standard microcontroller, not a custom APC ASIC of some sort. Hahahaha. I’m sure they’ll tell you that they put the stickers on for logistical reasons, but I think they’re also there to obfuscate. Anyone who knows anything about 8051 microncontroller part numbers will recognise that as an 8051 series 8 bit microcontroller. In this case, the P indicates the manufacturer, Philips (who renamed their semiconductor division as NXP) and the second C indicates 32kb of flash memory. I found, to my disappointment, that it was discontinued at the end of 2011, so I couldn’t get hold of a new one.

An 8051 microcontroller, in a post-2010 UPS? Why are they using it still? Well because if it ain’t broke, don’t fix it. APC have been building these UPSs on 8051 microcontrollers pretty much since the beginning. decades ago. and in the SMT series they added a second microcontroller, in my case a much more modern STM32 ARM Cortex M3 based chip, as a communications processor which interfaces with the main microcontroller via UART and handles external communications and the LCD front panel of the unit. They finally upgraded to more modern, powerful devices in the SmartConnect series I mentioned earlier, but there’s a downside. One of the key advantages of NOT having a SmartConnect model is that with these older ones, you can still hack into the UART communications between the two processors and adjust settings like battery charge voltage, even though they are not supposed to be user accessible, because the 8051 internally uses the same UPSLink protocol that APC’s older UPSs used externally.

power, supply, repair

But wait, there’s more

I managed to get hold of a document from Atmel (now owned by Microchip) that cross referenced the discontinued NXP part number to an Atmel drop-in replacement. the AT89C51RC2. which was still in production and available! And since this all happened well before COVID and the semiconductor shortage, I was able to actually get hold of one. Lol.

Getting a replacement microcontroller is easy, and soldering it in is just a matter of logistics. The real problem for me was the firmware.

APC offers firmware upgrade files for this unit, but they’re not the sort of thing you can just chuck straight onto the flash of the microcontroller and expect it to work! Through a process of research and experimentation, I was able to figure out that the firmware files APC provides are encrypted, and contain firmware both for the STM32 communications processor, and the firmware I needed, for the 8051 main processor. The communications processor runs a bootloader that accepts the file from a host computer over a USB or serial interface, and decrypts its own firmware and updates itself, before sending new firmware to the main processor as well. I could see, inside the file, a 32kB block of data labeled ‘MCU 05.0’ for the main processor. but it wasn’t in readable format. Long story short, it was a dead end. I’m no hacker (well, that’s what they all say isn’t it) and I didn’t want to get lost in countless hours of trying to figure out how APC encrypted their firmware files.

So instead of being a software hacker, I committed intellectual property theft (well, it’s pretty minor isn’t it) with hardware. I’d already bought a second UPS of the same model, so at the risk of stuffing that too and losing everything, I plugged my very untrustworthy programming setup into the working UPS’s microcontroller and purloined the firmware right out of it. That’s right, because they forgot to enable readout protection. APC, if you’re reading this, given that you’re going to the extent of encrypting your field update firmware files, you really should enable readout protection on your chips. Seriously. 🙂

But I’m not complaining. I managed to corrupt the firmware on the STM32 communications microcontroller as well, and used exactly the same trick to get that back too. because they didn’t enable readout protection there either. Nice.

The repair

It should have been easy from then on, but it wasn’t. I bought a replacement microcontroller, and went through the laborious process of soldering it in (not easy when it has 44 pins and I have only one soldering iron and two hands). I can’t say the soldering looked neat. I damaged many pads in my various attempts to solder it on and that means that the final result involves many ‘bodges’. However, after closely inspecting everything to ensure there were no dry joints or shorts, I was fairly satisfied that it wouldn’t fail on me.

power, supply, repair

I flashed the firmware, this time using standard microcontroller programming tools and the un-encrypted firmware image I’d obtained. Low and behold it worked. Well, that is to say, the microcontroller worked. I was lucky that the EEPROM chip, separate from the microcontroller, survived the entire process with its calibration values intact, however in the general repair process I managed to screw up some other parts of the UPS and had a lot of work to do in fixing it up.

I went through a long and gruelling process of tracking down some complicated problems with the 24V DC to 230V AC inverter, during which the main 12V SMPS of the unit decided to call it quits and produce 25V on the output! This didn’t help matters: it fried almost everything on the 12V rail, i.e. a good handful of ICs and ASICs, some of them proprietary. I’d fixed the original problem, but at the same time, completely botched up the rest of the repair! Also, I was using the working UPS I had for comparison purposes to troubleshoot problems, and managed to stuff that up too. soldering a probe wire onto a SMD resistor caused that to fail due to mechanical stress, and I then desoldered an IC to troubleshoot the problem and killed it due to ESD. While testing the broken IC in-circuit, the inverter operated incorrectly and I suspect inductive spikes from the main transformer killed one of the MOSFETs in the H bridge, because nek minnit there was a hell of a lot of smoke.

However, I was able to replace everything with the help of some Asian eBay stores which somehow had supplies of ‘genuine ,high quality ! [sic]’ replacements for the custom inverter ASICs. I theorise that they had gotten hold of rejected (e.g. due to high failure rate) batches of those ICs from the OEM, because several of the ICs I recieved were open circuit on all pins while some worked perfectly. Suffice it to say, I just found ones that worked and soldered them in, and fixed all the other problems.

With all that done, there was a heroic moment when the thing finally produced a 230V sinusoidal output for the first time in several years. Soon after, it gained enough trust to be plugged into the mains(!) and some time after that I considered it fully fixed, and my second one got fixed during that process as well.

Common Causes of UPS Failure

UPS failure is a leading cause of unplanned power outages in data centers. It results in costly downtime service and disrupts the overall operation of data centers. Batteries are the major culprit of UPS failure. Every UPS system has a set of batteries. One battery failure can lead to the failure of the entire set. Essentially, without the batteries, the UPS system cannot function.

Fans also play a major role in UPS failure. Most UPS systems have several fans. If these fans do not function properly, the system will overheat and shut down. Typically, fans can last for seven to 10 years. It depends on how frequently they are used. It can also vary with the temperature that fans work with. Note that the temperature they work with should be around 22°C to 25°C (72°F to 77°F).

Capacitors are also essential to the operation of UPS. They filter out the voltage spikes. If capacitors fail, the UPS system will not function. Typically, these components also last for seven to 10 years. Just like the fans, they also depend on the temperature they work with. If the temperature is higher, the capacitors will likely shrink immediately.

Here are a few more causes of UPS failure.

UPS failure is sometimes caused by power outages that last longer than the battery of UPS.

It can be that the old batteries were not replaced when the power outage occurred. Hence, the UPS system does not have enough stored electricity to continue operating.

Sometimes the UPS is placed incorrectly into the bypass. As a result, the fuses are blown. The rectifier fuses or inverter fuses can also be blown.

Overloading happens when the grid power requires more electricity than the UPS system could provide. This leads to the internal bypass of UPS. Meanwhile, underloading occurs when the power supply given to UPS is less than the power supply it needs.

AC power and DC power capacitors must be replaced regularly. The rule of thumb according to manufacturers is to replace capacitors every five to seven years.

The built-up pollutants inside the UPS can cause components to overheat.

This can lead to the system overheating and shutting down.

Like any other device, a worn-out and old UPS system cannot function efficiently. Overused UPS systems can lead to the failure of electrical components.

Prevent UPS Failure

Sometimes UPS systems automatically change to on-bypass alerts. This is typically caused by external or internal issues or no reason at all. Due to this, on-bypass alerts are usually neglected. But this should not be the case for the operators. In this mode, the UPS fails to secure IT equipment against utility power failures. The backup power is not operational. Hence, it is essential to monitor the working model of UPS. Do the necessary adjustments to put UPS into its necessary working mode.

Look out for the batteries. Failed or weak batteries also need immediate action. A battery set that operates at eight minutes may function less over time. Working only for five to six minutes. Sometimes it may not function at all without any warning.

power, supply, repair

Failures of lead-acid batteries are a major cause of UPS failure. UPS batteries should always be monitored even if they are relatively new. Batteries should be tested at least every year. If the batteries are not monitored properly, they may end up not functioning at all.

Monitor the air filters as well. Dirty air filters can lead to overheating and weakening of internal UPS components. So be sure to clean air filters in the facility.

Make sure that you maintain the right temperature in the storage of UPS systems and batteries. High room temperatures can lead to cooling failures. As well as overheating of UPS systems and equipment downtime.

Prevent UPS Failure with AKCP

AKCP Battery Monitoring Sensor

Monitoring is one of the best measures to prevent USP failure. For your data center monitoring needs, visit akcp.com. AKCP was established in the USA in 1981. With more than 30 years of experience in professional sensor solutions. AKCP created the market for various monitoring solutions in data centers. Today, AKCP is the world’s oldest and largest manufacturer of sensor solutions.

Battery Monitoring Sensor

Monitoring the voltage, temperature, and current load for batteries and DC power systems. The Battery Sensor monitors an individual cell or bank of batteries. As well as DC power systems like solar panel arrays. With this sensor, you can monitor:

  • The temperature of the battery terminals
  • The voltage output from batteries or panel array
  • Current load on batteries or charging circuit

Wireless Battery and Solar Panel Sensor

Wireless Battery and Solar Monitoring

Wireless Tunnelâ„¢ radio module with battery monitoring sensor can check battery health and solar panel output. And check solar panel efficiency. Monitoring current draw of batteries to ensure charging current is sufficient. The sensor monitors Voltage, Current, and Temperature. Works with all Wireless Tunnelâ„¢ Gateway.

Sensor Details

  • 4x AA Batteries
  • External 5VDC or 12VDC power
  • LED indicators for power, status, and RSSI
  • Optional DIN rail or pipe mounting

Typical applications would be for monitoring a battery bank, solar panel array, or generator starter batteries. Visit akcp.com for more details.

Power Monitoring Sensor

The AKCP Power Monitor Sensor gives vital information and allows you to remotely monitor power. Eliminating the need for manual power audits. As well as providing immediate alerts to potential problems. The AKCP Power Monitor Sensor is specifically designed to be used with AKCP sensorProbe and securityProbe base units.

It has been integrated into the sensorProbe and securityProbe web interface with its own “Power Management” menu. Allowing multiple three-phase and single-phase Power Monitor Sensors to be set up on a single sensorProbe or securityProbe. Depending on which readings are required. Visit akcp.com for more details.

Conclusion

Electricity is the ultimate fuel of data centers. Like any modern industry, data centers operate with a power supply. A secure power supply is essential to every IT facility. Power interruptions can have a significant impact on the operations of a data center. It can damage the most delicate IT equipment, leading to costly downtime. Backup supply then is vital to secure and prevent damages to IT facilities.

UPS failure has a significant impact on the functions of the data center. That is why it is important to monitor the different components of UPS systems. Note that the batteries are the most important component of UPS. When the batteries malfunction, the whole system cannot operate. Always monitor the room temperature where the UPS system is located. Do not forget the regular maintenance and monitor on all parts of UPS to prevent failure.

Leave a Comment