A Real-Life Breakdown Story
This is a fresh and painful account of a recent breakdown we attended.
A regular customer of ours has new Form Fill Pack machines installed in a brand-new facility. The oldest machine hasn’t been in operation for more than 12 months.
With three lines and a lot going on in getting the factory off the ground and performing as intended, it’s mayhem. Yesterday morning, the newest of the machines had an issue.
The first issue resulted in the picture below. Now, I didn’t attend this one and am not sure exactly how this happened, but everything went downhill from here.
Resetting the Initial Problem
Resetting this problem was reasonably straightforward. To do this:
- Move the drives to manual mode to release them.
- Move the drives to the home position.
- Straighten any bent bits back to their original position.
Then, home the machine, and away you go.
Only it didn’t go.
Operator Intervention and Escalation of Problems
The site engineers were called away to another issue. While the operators on this machine were cleaning up after the crash, the engineers left to sort the other issues and planned to return to this one shortly, with the machine cleaned and ready to start.
Unfortunately, they didn’t return quickly enough.
The operators faced another issue and decided themselves that the best course of action was to power cycle the machine. However, when you power cycle machines with lots of electronics, you need to give them long enough to discharge. Otherwise, it could cause additional issues and confuse the drives, PLC, etc.
This led to additional problems and spiraled out of control. You can easily get into a position where you have misleading faults, taking you in different directions.
The operators didn’t leave it long enough to power down properly, so the engineers powered it off again and left it for 20 minutes.
Arrival on Site and Initial Diagnosis
This is when we arrived. Upon powering the machine on again, we noticed an error on the DC Bus: Allen Bradley Kinetix 5700 series displayed the error code FL2 Conv Overtemp.
Now, in all honesty, we hadn’t worked on these drives before, so we didn’t have the experience of knowing their quirks.
Additionally, there were a lot of drive errors on the HMI screen.
Challenges with Diagnosing Servo Drive Errors
One of the problems with machines that are servo motor-driven and have a comprehensive safety system is that when a fault occurs, generally, the HMI screen shows all the drives in fault. To diagnose the issues, you need to access the PLC program via Allen Bradley Studio 5000.
Unfortunately, we didn’t have access to the program in this factory. One reason for this is warranty concerns and the theory that the OEM will assist in fault-finding during the warranty period.
While this could be the case, the delay in getting the OEM software engineer online caused many additional hours of stoppage.
Initial Attempt: Replacing the DC Bus
Given the information we had, we decided a good place to start would be assuming the DC-Bus had failed.
We proceeded to swap this with a spare Kinetix 5700 DC Bus, which takes a while because of the IP address configuration.
So, we swapped it out.
The fault was still there.
Doh!
Identifying the Root Cause
Engaging our brains, we realized that this fault was indicating an earth issue, and the most likely cause was a short on a motor or the cable assembly.
By this time, the programmer had got online from Germany. This didn’t help much from the program side as he could only see that a multitude of drives was in error.
So much for the advantages of using servo drives and Computer Controlled Logic!
I was longing for an asynchronous motor and a DOL start contactor—it would have been up and running within the hour!
Manual Investigation and Resolution
We had what we had, and all we could do was manually disconnect the drives, check through the connections, and look for water ingress.
This can be a lengthy process. Additionally, it’s not as simple as disconnecting the cables and seeing if the fault is removed. The machine needs to be powered off each time; otherwise, there is potential to damage the encoders. This adds quite a bit of time to the process.
The Fix
After carefully working through the issues, we isolated the fault to a specific motor and resolved the problem by replacing the faulty cable assembly. The machine was finally back in operation after hours of troubleshooting.
Lessons Learned
This breakdown highlighted the importance of proper shutdown procedures and having access to diagnostic tools like Allen Bradley Studio 5000. It also underscores the value of skilled engineers who can methodically work through complex issues.
For more stories about how we troubleshoot and solve industrial automation challenges, check out Our Latest Work.