Network Security at a Hydro Plant pt. 1

The small hydro industry is pretty old fashioned. Many of the people working in it come from old industrial backgrounds or have been doing hydro for a long time (or both). I mean absolutely no offense by this, it's just that you don't really fall into small hydro by accident. It's not a entry level job out of college and most people have never even considered that a private individual could own a hydro power plant.

Through no fault of their own this typically means that from a computer/technical side the small hydro industry is pretty far behind. If you go to the hydro conferences (which I did) you don't see anything about network security, remote access, or even the HMI/SCADA systems that almost every plant has. Frankly I was surprised to see almost nothing regarding PLCs and automation at the hydro conference. There were vendors who would give you "water to wire" which it appeared would also include a PLC and HMI/SCADA but to call it secondary would make it seem more prominent than it was.

So we've been acquiring new plants which are badly in need of some TLC and many of our old dial-up remote access systems are now failing. This has lead us to look at how we can put our plants online (the internet) without needlessly exposing them to hacking threats. If we're online there's always the possibility of hacking so we want to get the best security possible. The challenge was that we're securing multiple sites (11 at the time of writing this) and we're still small business size so we can't invest a ton of money in each site. Think about it, if we had only one location to secure with all our infrastructure it would be pretty straight forward, but we have no infrastructure and 11 separate sites to secure.

The Cyberoam CR-15wING Unified Threat Management appliance

Cyberoam CR-15wING

After some shopping around and experimentation we settled on the Cyberoam CR-15wING Unified Threat Management appliance. Cyberoam is a smaller player in the network security world (than CISCO) and they're currently big in Europe and Asia and still breaking into the US market. After my frustration with CISCO's ISA-550W and it's subsequent EOL (we can talk about that later) I set out to find a solid network security appliance with reasonable cost and support contract. 

Enter Cyberoam. On a whim I contacted them about our needs and how they could fit in and immediately got a call and an offer to have an evaluation unit sent to me immediately. After checking their prices (which were very reasonable) I agreed. I need to confess to not being a network guy so I don't want to pretend to be an expert, but I was quite overwhelmed by the CR-15. I can only compare it to the CISCO ISA-550W which was a small business network security appliance priced similarly. The Cyberoam interface was significantly faster and more stable than the CISCO. It appeared to be much more powerful as well. It was certainly more complicated for someone like me to deal with. Had Cyberoam technical support not spent three hours on the phone with me walking me through the setup I would have given up and sent it back. They were very patient and in the end I came to understand how much more secure then new SSLVPN was vs the CISCO IPSEC VPN. 

I'm also beginning to understand more about how to segregate our control systems from the regular intranet at our sites. By restricting access to the control systems we can further prevent intrusion. I'm still working on understanding everything about the Cyberoam device but for our small hydro plants I believe we have a solid platform with a lot of room to grow. 

Look for part 2 with some more information about how we are securing our sites. 

Utility Belt Prototype

So, aside from hydro, I've been working on designing my own bags and utility belts. Back in July I finished my first prototype, but never blogged it.

The utility belt I designed and created in July was created specifically to allow me to carry a few specific items around with me easily while working:

• Phone (sized for the iPhone 5 carried horizontally)
• Flashlight (AAA size)
• Knife (specifcally the Leatherman CS4)
• Pen (of some sort)
• Pack of 3x5 index cards for notes
• Something else (undesignated pouch)

There were a bunch of other items I had that I was playing around with but those were the most important. You can see some of my design ideas here.

The second from the top was chosen to be the first design I made.

I also spent a lot of time trying to decide on and then acquire the appropriate materials. I settled on waxed cotton canvas and wool as my main two materials. Waxed cotton ended up being very difficult to find in small quantities. Eventually I found someone on Etsy who was willing to sell me two yards of grey waxed cotton, which I'm not even close to being done with.

So it took me a good deal of time and experimentation, but back in July I produced my first prototype. I've been wearing it nearly every day since and I'm happy to say that my design was sound. My technique has continued to improve but ultimately it was a wild success as a prototype.

Prototype utility belt after six months of use. 

You can see the finished product here after I've been wearing it for about six months. I wasn't very good about finishing the edges so there's some fraying but all the seams are currently still solid.

So the design I came up with started with the belt. The belt is made from a backing of 1/16" industrial wool felt with 1.5" seatbelt nylon as a middle layer. A 1" nylon top layer holds the Cobra brand 1.5" buckle. These buckles are really sweet but quite expensive. They're technically overkill but I was set on using them and they're great. This belt is solid and doesn't slide around because of the extra friction offered by the wool bottom layer. It is somewhat adjustable (~4") allowing me to wear it over jeans or over insulated coveralls.

The pouches here are the same shown in the drawing and worked out quite well. At the time of construction I added another small pouch for soap stone (used to draw on metal). I used waxed canvas for just about everything but added on some grippy black material I found at Ragged Mountain onto the pockets and a contrasting yellow to the inside. I decided to use strong magnetic clasps in the smallest size I could find. I got them from ... and they ended up being perfect. I didn't do a good job of reinforcing the flaps so I added extra stitching later which looks strange, but it works. The trick with the pouches was to use the industrial wool felt as the backing for them. The belt goes through the pouch between the felt and the canvas so the felt of the belt is against the felt of the pouch providing excellent friction. The keeps the pouches from sliding around your body in general use without making them hard to move it you want to move them. It was an idea I had that ended up working perfect.

One idea that didn't really work well was the universal pouch that I added. The idea was to have a pouch that could hold a variety of things but ultimately it couldn't. I even added an extra long flap with a second magnetic snap so you could fasten it looser or tighter. It was a good idea, but the pocket itself was not well constructed. I didn't know how to make good pockets and things like falling out of it. Additionally it was just the wrong size. Nothing really fit in it well other than a granola bar. Oh well.

Ultimately I learned a lot and I'll post some of the other bags I've made since.

Prototype in progress.

When your dam is empy on the inside its a ...

Hollow Dam.

Hollow is our latest project. It's another old Algonquin site out near Gouverneur, NY. A group of us took a nice drive out there this week to look at what it would take to give the site a control upgrade much like we did at Burt Dam. The weather held out pretty good for us and we got a lot done.

You can see the "open" turbine at the left. 

The site has some very unique turbines. They are vertical cylinders that contain the turbine and the generator and act as a gate. The entire cylinder raises up by about three feet which opens the "gate" allowing water to flow into the cylinder and through the turbine. The turbines themselves have variable pitch blades allowing the operator to regulate the water flow and power generation to some degree. Unlike most of our plants, there's no real powerhouse around the turbines, which makes winter repairs rather annoying. Beyond this, though, they seem pretty reliable and produce about 1 MW/h (1000 KW/h) at peak. The two machines are identical and are rated at 530 KW/h each but tend to max out around 500 which is still good.

This is the rig around the two turbines to allow for maintenance.

Because of how the turbines are configured, the control room is an entirely separate building down below the dam. It's a bit cramped but nice and quiet, unlike most of our plants where the controls are in the same building as the turbine. The control panel appears to be similar to a lot of the plants I've seen recently. It was upgraded in the 80's and from there the bare minimum was done to keep it running.

The PLC is an old Toshiba model (same as Burt) but is not functional at all. Instead of spending the money to get it working it was bypassed as much as possible and is now run only on manual control. For Hollow this is okay (not great) because the operator lives on the premises. The previous owners wouldn't have gotten away with that any other way.

Kinda cramped, but quiet and warm. 

Richard and I went through the panels and started looking at the electrical drawings. The panels are pretty cramped too, having been pretty efficiently designed to be as compact as possible. You can see the two control panels and their cooresponding breaker cabinets in the picture. Out of frame is another control panel primarily containing protective relays along with a few non-functioning switches and the PLC cabinet which is near the floor.

Unfortunately there isn't any good space on the panel for one of our touch screens. At this point I'm thinking that we'll probably build a mount for the screen and put it on the desk which is at the left of the picture. I really like the Beijer panel we used at Burt and the operator likes it too. The operator at Hollow is less technical then the one at Burt and I don't think putting a full PC there makes sense. We'll probably use a wifi android tablet or iPad for him to keep an eye on things from his house. That will also give him access to the reporting tools we're setting up.

Overall it looks like a good little project. I'm not exactly looking forward to two weeks in Gouverneur (or maybe Watertown), but otherwise it looks like fun.

... so why is it called "Hollow Dam"? Well... we found/saw the remnants of the original dam...

Kinda looks like it was empty on the inside...