Haha , you thought PC's were hard ?!!

[BrainEater]

In the OP it wasn't clear if this PLC/microcontroller problem was caused by or co-incidental with the mechanical problem you mentioned. The filler has to have black start capability and if the PLC and connected controllers are all live and communicating, it has to be a synchronization issue. Some transducer is out of sync.

Just a thought.

P.S. I'm assuming it was running previously, and I know you should never assume anything.:thumb:

You are not far off . (and no , it does not have 'black start'!!) ((Sheeeesh))
Now that it's fixed , I can tell you the exact failure tree.

---------

:thumb:WWWOOOOOOOOOOOOOHOOOOOOOOOOOOOO IT"S FIXED !!

The magic was only partially mine , but we have avoided the 10,000$ step !!!

----------

So this will be a wall of text but I can now explain.

When a bottle enters the filler , basically a 'gear' (called a starwheel) feeds a bottle onto a bottle sized platform on the filler....this platform then pushes the bottle up to a rubber 'bell housing"....this actually centers the bottle in relation to the fill tube , which comes down into the bottle from the top.

The issue we had on Monday was that the rubber bell bell housing stuck up , so the bottle was not centered.The fill tube then came down , and managed to exactly jam on the lip of the bottle.The machine then hard stopped , with all valves stuck in current position.
This allowed beer to leave the bowl (holding tank) and co2 to openly jet through the now empty top of the bowl.
The co2 cooled the top bowl temperature gauge past it's limit (low range error)

At this point the serial interface module for that sensor lost it's mind.
It corrupted every single device down the Ethernet chain , right into the PLC.
----
What a week , I'm getting drunk right efffing now !
woop !!!

:thumb:

 

[SugarJ]

Congrats on getting it sorted!

 

[Dzzope]

Ohh man that sucks so hard... Need a catch put in there for those (alarm, sensor whatever)

Though at least you know to check the line for possible mechanical failures if it happens again rather than just the control systems.


Taking some produce home tonight or are you not going to make it?

 

[Valkyrie]

You are not far off . (and no , it does not have 'black start'!!) ((Sheeeesh))
Now that it's fixed , I can tell you the exact failure tree.

---------

:thumb:WWWOOOOOOOOOOOOOHOOOOOOOOOOOOOO IT"S FIXED !!

The magic was only partially mine , but we have avoided the 10,000$ step !!!

----------

So this will be a wall of text but I can now explain.

When a bottle enters the filler , basically a 'gear' (called a starwheel) feeds a bottle onto a bottle sized platform on the filler....this platform then pushes the bottle up to a rubber 'bell housing"....this actually centers the bottle in relation to the fill tube , which comes down into the bottle from the top.

The issue we had on Monday was that the rubber bell bell housing stuck up , so the bottle was not centered.The fill tube then came down , and managed to exactly jam on the lip of the bottle.The machine then hard stopped , with all valves stuck in current position.
This allowed beer to leave the bowl (holding tank) and co2 to openly jet through the now empty top of the bowl.
The co2 cooled the top bowl temperature gauge past it's limit (low range error)

At this point the serial interface module for that sensor lost it's mind.
It corrupted every single device down the Ethernet chain , right into the PLC.
----
What a week , I'm getting drunk right efffing now !
woop !!!

:thumb:

It's great that you got it fixed for relatively cheap. What I meant by "black start" was controllers functional and able to communicate AND nothing jammed, bent, broken, or otherwise out of whack and it should be able to power up and figure out that it's OK. Were the controllers able to corrupt the chain of command to the point it could not do that when the mechanical problem was cleared? I used to clean fillers long years ago, before I changed to the technology path. Never actually worked on them as a tech though.

It's always nice to be not far off, but it's better to be right on. Cheers!:thumb:

 

[BrainEater]

Yea , to me, black restart=OS reset , go !

No, the machine was totally bricked. Controllers hardlocked and unable to start, and some stuff not communicating.

It started with an analog (4-20 mA) temperature probe going below range (3mA) ....They don't like that !!
The analog interface module then promptly lost it's shit and cleared it's sensor bios. This then caused the serial interface module it was plugged into to forget all of it's "communication protocols"....
Then it went down the chain and corrupted 4 more manifold controllers (5 out of 7).
Then it took out the PLC proper.
------

I am not an automation expert by any means , but I've NEVER seen a failure like that.
It took a real automation specialist (cheers J) , 3 days to get to the bottom of it.

Seems to be workin , 2nd run tomorrow.

:thumb:

 

[sswilson]

It's downright frightening that a catastrophic, unrecoverable failure could cascade like that in a large scale operation. You'd think that at some point the next piece in the chain would just say "that ain't right" to the input it was getting and just automatically shut down as a failsafe.

 

[3.0charlie]

And in Aircraft System design, that failure analysis is called DFMEA - Design Failure Mode and Effect Analysis. https://en.wikipedia.org/wiki/Failure_mode_and_effects_analysis Basically for each part of a system, if it fails, what will be the effect.

Quite interesting.

Now bring on the beer.

 

[BrainEater]

Roger that, bring on the beer indeed !!
The machine is running again, and happy.
-----------------

The only reason I care about failure mode analysis is so I can justify and explain the giant bills, when I gotta pay for it.

This kind of work should be the manufacturer's job, not mine.
When I called the techs, it was pretty obvious they were simply reading down a "troubleshooting checklist".They asked me 10 times if the fracking air was connected , and would not let it go....I reached a point where I said "If they ask me about the air again , I'm flying down there to choke the shit outta somebody!!!".

I stopped calling them, and called in a third party bona-fide-professional.It took him less time to figger out step 1 , than it took the tech how to remotely connect his own software FFS !
....And it cost 1/4 of what their man wants just to show up.

With last weeks crash course on 'rotary bottle filler programming' I am confident I will only need to contact the manufacturer for the most extreme issues.Which is good , cause they are trying to bend me over the couch.
*The actual fix , took less than 5 mins of work* (not counting 3 days of reverse engineering.)

 

[Valkyrie]

So were you able to improve the fault tolerance of the control system in this process? It's amazing to me that you had to engage in reverse engineering on a half million dollar product. The tech industries I worked in demanded full documentation, or no sale, no exceptions.