I got a call one day that some new floor scrubbing robots being trialed in the field would get stuck for 15 or 20 seconds even after an obstruction was cleared.
These new robots were outfitted with a camera tower so they could scan grocery store shelves as they autonomously scrubbed the floor of the store. The SW team said they could see the Ethernet connection disconnect between the main computer and the camera computer.
A sporadic event
The Ethernet drops were hard to replicate as it only happened several times a day at seeming random times.
I was finally able to find a robot that had a lot of dropouts and had that moved to the lab. I set it up with an oscilloscope with a differential probe watching the Ethernet signal and tried many things to see if I could get the Ethernet to disconnect.
Finally reproduced the issue
I finally did it. Here’s what I saw. Here’s a normal scope plot (Tektronix heatmap enabled):
Normal Ethernet 10/100 signal
But when the emergency stop was released, there were sudden bad incursions into the eye diagram, just extreme noise.
Sudden Noise on Ethernet 10/100 signal
And then the two computers stopped talking to each, leaving only the sync signal. It’s like the communication got so garbled that they just disconnected from each other.
Disconnected Ethernet 10/100 signal
After some time puzzling over the schematics of the two computer systems I spotted a tiny mistake, so easy to miss.
The connection between the two computers can be simplified to the schematic below. Can you spot the problem?
Simplifed connection diagram between Main MCU and Camera MCU
It’s so small most engineers would probably miss it. In fact, each computer would likely pass all testing of their ethernet ports.
So here’s the problem: the Main CPU’s ethernet transformer’s center-tap is grounded to electrical ground, but the Camera CPU has its ethernet transformer grounded to chassis ground (the correct way). See the different ground symbols in the schematic above?
This would cause common-mode noise to get coupled into the Main MCU’s electrical ground instead of being shunted harmlessly into the machine’s chassis ground. And it turns out coming out of emergency stop causes a huge noise spike in the machine because the motor controller closes a relay and applies 1,500 uF of capacitance to the battery! (More of that in another post…)
So this problem only happens when:
- the two systems are connected together, and
- there is a very large electrical noise event
A field workaround
A big problem with updating the Main MCU board was that we already had thousands of machines deployed in the field. Reworking thousands of units was out of the question.
I had an idea to use an isolation cable to galvanically isolate the two sides of the Ethernet connection and basically isolate the main MCU externally.
To quickly test this theory, I wired up a cable that had an extra Ethernet transformer in the middle. The idea was that this cable can be installed only on the few problematic machines in the field.
Proving the issue with a cable solution
And it worked! The cable provided isolation between the two systems and robots that previously had Ethernet dropouts no longer had the issue.
While off-the-shelf isolators were readily available, they mostly had RJ-45 connectors. The cabling we used had metal industrial locking connectors. This simple PC board let us cut an existing cable in half, solder the board in, and heatshrink the whole thing and send them off for installation.
The problem is still out there
The problem will probably continue to persist because a few designs will ground Ethernet incorrectly.
Here’s part of a schematic for an evaluation board from a major IC manufacturer that makes the same mistake. Granted the board is to demonstrate a PoE DC/DC converter, but an engineer might see this schematic and follow the same grounding mistakenly.
Example from an evaluation board from a major IC manufacturer
Engineers reading this will hopefully learn from this example and do it right. This was a great example of debugging that truly got down to root cause.
