Glados, is that you?

So for my main development machine, I picked up a 2012 27" iMac and maxed it out. Great machine, I already love it. I took a bit of a risk though and decided to purchase the VESA compatible one and a custom arm for it. The arm I got is pretty heavy duty and will hopefully give me some extra flexibility in my ergonomics. I have to say, it is quite impressive. Much thanks to ErgoMart who accidentally sent me the wrong one but them were great about handling it.

Debugging a Model A

I got a call today from my father. Turns out that he was having a fuel problem with his new Model A Ford hotrod. The thought was that it was probably out of gas, but he wasn't sure because his fuel gauge didn't work.

He basically built the Model A from scratch. He pulled the rusted out frame and body from the woods and rebuilt it from the ground up. It's been cool watching him build it up from nothing but I must say, it's incredibly complicated to rebuild a car, even an old one. I sometimes imagine trying to rebuild the van from the ground up and it's mind boggling. The Model A is a 1930's car so it's not quite as bad. I can see now why Peter D has taken 8+ years to rebuild his 60's era Beetle.

So back to the task at hand. Putting more gas in the tank did NOT help and we had to assume that the fuel filter was clogged. A quick trip to his house yield s a fuel filter from an earlier vintage that should work fine. swap out the filter and he gets about another mile further along before it dies again.

This time we find that the fuel line is clogged and the filter seems fine (the new filter was in a transparent housing). So a wire up the fuel line and a drain pan was necessary. And, low and behold, the fuel line was clogged up with ... bugs. That's right, the fuel tank was full of dead ants and metal shavings. Well, not full, but enough to clog the fuel lines a few times. We babied it home and my father spent the rest of the day cleaning it out. yuck.

I wish I'd have gotten picture of the actual bugs. It brings new meaning to the term debugging.

More old mill stuff

I spent some more time exploring the mill the other day. Found some more cool stuff I wanted to post.

HELP!

Sir, we've got a leak in our penstock.

Are you feeling lucky, Mr. Bond?

For the record, this thing is very big and very rusty and if it fails there will be a very big problem...

#2 PLC issue revisited

So after we got back from our field trip, I stopped in at the Gilman mill to (hopefully) start moving our desks to the office. Turns out the cargo elevator is still dead so we STILL (at the end of week 3) don't have anything in our office. -_-

Anyway, while I was there I was talking to my partner and his father and the topic of the glitch with #2 came up again. It would seem that the "stuck button" problem has happened again in just a week. My partner's father held nothing back in voicing his frustration with how the PLC was now dependent on the PC and how F*&%ing stupid that was. I had learned enough about InduSoft the previous week to feel comfortable adding in a couple of status indicators that would show the current state of the virtual relay in the PLC. If it works the way I think it will, they operator should be able to see if they relay gets stuck because the indicator light will stay on. Hopefully I'll get an update on Monday as to whether it helped.

I have a few ideas on things I'd like to change but I'm still a little reluctant to start reprogramming our PLC...

Field Trip!!!

So my partner, Greta and I took a field trip to a great little hydro site in Wapinger Falls, NY. Harry and Sarah made us feel incredibly welcome as they walked us through all of the cool stuff they're doing. They own three sites and have really taken great steps to make everything consistent and usable. Their focus was on automation and remote monitoring and they've really done a fantastic job.

The first thing Sarah showed us was the more than 16 networked surveillance cameras that they can access right through their iPad. These cameras are set up at all their sites and are a combination of static cameras and Pan/Tilt/Zoom (PTZ) cameras. The PTZ cameras they are using are top-of-the-line sony cameras with 26x optical zoom which allows them to use one camera to monitor both overall spaces and individual gauges on a panel! Very nice setup and something we're looking to do at our sites as well.

The Wapinger conference room. PHS has got to up it's game!

The next thing Sarah showed us was the PLC control panels they had put in. Instead of a PC running something like InduSoft, Sarah is using a HMI touch panel to interface with the PLC. Their system is from Beijer Electronics and is a stand alone hardware/software solution that includes networking, remote access, reporting and control. These touch panels are completely customizable and programmable via C# which is appealing.

Two of the Beijer panels from two different sites. Each panel is customized to look like the machines at the site. 

Turns out that PLCDirect has their own panels now called C-More HMI Panels (Get it?!). These panels would theoretically be designed more specifically to work with the PLCDirect PLC but they don't look as high quality and there's no information on how they're programmed. PLCDirect's competitor to InduSoft looks, if possible, to be even worse. It's possible that the C-More panels would be worse as well. I need to do more research.

Harry and Sarah were awesome to us and the Wapinger site is really a show piece. There's an onsite apartment, a party room with dance floor, salad bar and soda fountain and all sorts of neat touches. I'd love to turn one of the PHS sites into a show piece.

The network closet wasn't a show piece though...

Whereas this old panel (not in use) really was!

After Wapinger, we all went to Leppert-Nutmeg to see how generators get refurbished. These guys had just refurbished one of our generators and we thought we'd stop in for a tour. Unfortunately the guy who was supposed to give us the tour was busy. Instead, the president of the company grabbed us and gave us an amazing tour of their facilities. We were very impressed, not just by the breadth of services they offer, but of the specific knowledge that the president of the company shared with us. He knew about every project in there, what was being done and what state it was in. He shared some mind boggling stories about some tidal generators that were never designed to be serviced... Turns out they needed to be but it's just not possible. Everything had to be scrapped instead of repaired.

I wasn't able to take any pictures here but it was awesome and I wouldn't hesitate to work with them again. 

My first day on the PLC

FINALLY!!! After nearly two weeks, I've gotten the opportunity to work on something that fits my skill set.

Turns out that we had a glitch in the control systems for our #2 generator in Gilman. Apparently, the wicket gate that controls water flow to the turbine would always re-open no matter how often you tried to close it. This meant that our PLC programmer would be coming up to take a look at it and see what might be wrong. I dedicated my whole day to shadowing him in order to learn as much as possible about our control systems.

Turns our there are two parts to the control systems in Gilman. The first part, the most important part, is the PLC or Programmable Logic Control. This is the computer that lives inside the control panel for each machine. It contains all the logic for when to open gates and all the water level, temperature and hydraulic sensors. In general, the PLC is pretty simple and typically very stable.

(This is not our PLC but is a representative picture. I thought I had one of ours. I'll replace this soon)

Each of the vertical pieces is a module that handles inputs or outputs. Inputs would be from sensors, outputs would be to relays. One of these is also the CPU and another is the communication module (Ethernet). I believe the Power supply is separate and there is also a bridge (between horizontal groups of modules).

Our PLC is a PLCDirect DL205 Series. Supposedly we've standardized on this series across all of our sites. There are always new ones coming out but standardizing is also a good idea. I don't expect to be upgrading these for a while.

So in order to program the PLC, you need to connect to the internal machine network. This is a normal ethernet network, but is not connected to the internet (yet...). We then use PLC Direct's DirectSoft 5 programming software. It is programmed using Ladder Logic, which is apparently an electrical engineering thing but seemed straight forward enough.

Not the most informative image, but this is what DirectSoft 5 looks like. This shows relays and what they trigger. The left panels are debugging tools.

So DirectSoft 5 outputs into the PLC so it never produces a UI for anyone to see (OH THE HUMANITY?!?!?). Instead, the physical control panel is the UI, which is kind of novel. So in addition to DirectSoft, there is a remote monitoring tool located in the main office that aggregates information from all the PLCs (one per generator) so the operators can see everything all together. That is the other part of our control system and was created using a tool called InduSoft Web Studio. To be blunt, it sucks.

A screen from indusoft showing the status of the #2 Generator

It's incredibly bare bones in terms of it's look. We're a version behind so maybe it's improved, but I doubt it. It's designed to provide a read-only interface to the PLCs. It has the ability to provide write access to the PLC but we've found it to be buggy as you will find out. 

So, if you look closely at the image you can see two buttons in the middle of the page labeled OPEN WG and CLOSE WG. These buttons were added for unknown reasons in order to allow the operator to open and close the wicket gates without having to leave the office (lazy?). These buttons perform a write operator that closes a virtual relay in the PLC. That virtual relay is a parallel path to the physical relay that will trigger the wicket gates to open. The button in InduSoft is supposed to be an instantaneous button so that when the mouse button is down it closes the relay (on) and when the mouse button is up then it opens the relay (off). The problem, after a good deal of debugging turned out to be that the virtual relay was stuck closed (on). The VIRTUAL BUTTON was stuck... By clicking the button in InduSoft a couple times we were able to get the virtual relay to open (off) but we could not figure out how the relay got stuck in the first place.

One of the problems with InduSoft is that their components are closed. So beyond setting a couple properties in the on-screen button you don't have any control over how the button sets the virtual relay. The best I can come up with is that there's a momentary communication hiccup when the user stops clicking the button and it fails to send the relay open signal. Once that virtual relay is closed, there's no explicit way to open it. This is a bad design as you NEVER want there to be something in the PLC that can only be effected by the computer. We'll need to fix it.

How do you know your hydro site is working properly?

Got trained today on at the Gilman plant. I use the term loosely because I'm not going to be an operator but I need to start understanding everything that goes on. My partner took my around and walked me through the "mechanical rounds" which the operators make in order to ensure that everything is running properly.

So how does one determine whether their hydro site is running properly? Essentially it comes down to checking about a hundred oil levels and temperature gauges. You see, if you run out of oil (or hydraulic fluid) then shit starts going south... fast. Similarly, a bearing that is going bad then it will generate more friction which will cause it's temperature to rise. I'm not 100% clear on how many of those temperatures are monitored via the PLC.

One of the reasons that Gilman has a full time staff, unlike all our other locations, is that it is not fully automated. One of my tasks will be to help bring it up to full automation, though I'm concerned that in doing that I may end up putting some people out of jobs...