This article begins a series of articles about a particular large system at UNC
We had a big problem at UNC. Our energy management control room was filled with different so-called enterprise systems. Our buildings were filled with every sort of low bid system that could be purchased under state rules. Where we had been able to impose a standard, internal incompatibilities, between systems from twenty years ago and from ten years ago supplied by the same vendor were as bad as those between product lines. The men in the control room were running a half dozen different incompatible monitoring systems, and all of them were fraught with problems.
As we built out the campus network, it was no longer acceptable to isolate the building systems onto separate webs of leased phone lines. As we moved the systems onto standard networks, the problems became worse. The first attempts by the building systems vendors worked only ion fully isolated networks. Many systems implemented quirky variants of TCP/IP that operated only with certain brands of router. No two chose the same brand of router, and none chose the brand of router selected by out enterprise networking group. With clever trickery, and great expense in time and labor, we were able to work around these problems.
It was surprising how badly the control protocols were written. Developed in isolated labs, by engineers who understood little about the challenges of sharing a network with unknown others, they were really quite bad. Not only were all aspects of these installation discoverable, but they were sensitive. I know one campus whose coal plant control system was taken off line each time an un-configured LaserJet was put on the campus network.
Enterprise control protocols of the day relied on security through obscurity. Anyone who could read BACnet, or LON could discover and operate the entire network. Proprietary protocols such as Metasys or Ultavist were no better. So again, through hard work, and no small expense in network equipment, and even greater expense in time and labor, we put in a complex network of VLANS operating inside the diverse business VLANS of the were able to work around these problems.
The real problems were much worse than I am letting on. Some systems would stop communicating if a router were switched out – until each system in the subnet and neighboring subnets had been simultaneously cold booted. Network stacks written by controls systems houses were abominable and there network interface cards were expensive. I remember tuning network ports down to 10MHz half duplex, to support a $1000 dollar Ethernet interface, while I was buying 100MHz full duplex network cards for $19.
With all the work, with all the effort, with all the off balance sheet costs, I knew the game had to change.