EBMS – Bringing the EBMS Project to life

This post continues a series about a particular large system at UNC

It is said, you go to war with the Army you have, not the Army you want. And you go to bid with the contractors you have, not the contractors you want. And you specify the technologies the market has, not the technologies you want. And we had to buy the Enterprise Building Management System (EBMS) we could rather than the one we wanted.

When we were finalizing the specification for the EBMS, Web Services was all the buzz among control vendors. Many vendors demonstrated SOAP-based building gateways to us – but the product never quite had a part number. Standards were non-existent. oBIX was not yet even a committee draft. The BACnet-WS committee had not begun to meet.

Administrative IT at Universities is always behind the times. The culture that cherishes and maintains Old East, the first State University building, preserves and protects old business processes as well. Our CIO had pronounced a year before that he was pretty sure that XML was a fad, and he knew Web Services were a fad, and we were to let none of that stuff in here. We needed innovation; we needed to push the building controls industry. Such efforts require very clear communication. We went to bid with a document that never once mentioned Service Oriented Architecture, or Web Services.

Building system contracts are traditionally managed as construction projects. UNC is a state agency. State construction projects rely on a very structured business process producing very predictable results. Because we were breaking new ground, we needed to contract for performance, not design. Performance contracting is not part of the normal process in the North Carolina State Construction Office.

State construction did not know what to do with this project. Our creative and resourceful energy manager, Jay Evans, continued to fight for EBMS. At one point, we actually received a suggestion to bid a small portion of the project and do most of the work on change orders; Jay rejected that approach and kept pushing. Finally, State Construction agreed that we could run a controls project through the State IT business process; SIPS supported an RFP / performance based project cycle.

So it began. All communications outside the building were going to be XML based, but no standard was specified. All Monitoring and Operation was going to be from a single web-based system. All graphics were going to be vector based, so every page could display on any device, and scale as needed from large screen to small. (And this worked – I’ve already seen our system operated from an iPhone).

Fortunately, every group associated with the final team approached the project as partners. An example of this occurred about six months in. To preserve legacy systems until the next building renovation, we were placing special purpose gateways in front of the existing building systems. The subcontractor installing these gateways, Yamas, was using the only solution available at bid time: a proof of concept web services engine available from Tridium. oBIX had become a committee draft, and was sliding into the slow process of OASIS Committee Standard approval. Based on a Phone call, Tridium agreed to re-write their gateways around oBIX and deliver us the newer product based on the standard. Yamas agreed to accept later delivery and still meet their delivery schedule to provide us a standards-based solution. Cyrus Technologies promptly agreed to revamp their development work our standards-based vision. All willingly cooperated as a team to deliver the innovative standards based solution we wanted.

One last system hurdle rose up in our path – standards-based XML graphics. When we launched the project, SVG (Scalable Vector Graphics) – a vector-based W3 standard - seemed the clear choice. During the project, the market leader in SVG tools and rendering bought a market leader in a competing technology; Adobe abandoned SVG after it acquired Flash. We had to scramble and switch to XAML. I am not entirely happy with that change, but that was the way the market developed. On the bright side, there are some DWG/XAML converters, so it looks like we will be able to bring our AutoCAD floor plans into EBMS.

Open Standards-based development is new to building systems. There will be hurdles that you do not expect. You need a team that is willing to commit to a vision. You will probably need to define a performance based project, as the road is not entirely clear. For us, the results are well worth it.

EBMS – Design Decisions for the UNC Enterprise Building Management System

This post continues a series about a particular large system at UNC

As described in the last post, Building System protocols suffer from a number of problems. They are unstable. The protocols are not standards compliant. They have poor security characteristics. So we decided to put them in a sandbox.

Each buildings system was to be isolated within that building. No protocols not easily recognized by Network Support Technicians would be allowed on the backbone; BACnet, LON, et al., were all gone, even their IP versions. All network traffic for control systems would come from protocol stacks that were written for mainstream use. Systems would be secured using our standard network security

All connections to building systems be sessionless. This one needs some explanation. When a program on your computer connects to a database, you log in, and establish a session. That connection is maintained by regular hand shaking. The remote server remembers things about you, so you can continue the conversation. This is efficient, but it does not scale well.

When you contact a web server, you connect, get your page, and then the web server forgets all about you. This allows one web server to service thousands of clients. The standard for sessionless program to program communications is web services. You use web services whenever Gmail types ahead to finish an address for you. We opted to use web services for all routine communication.

We defined two ways to connect to the sand box. It was necessary that we choose names that were not tied to any particular brand, so we made up our own. All communications between the EBMS and the control system are through the Enterprise Building Local Gateway (EBLG). All maintenance of the building system is through the Local Control Station (LCS). Both access points are secured by network security.

We defined a Local Control Station (LCS) to support the needs of each individual system. The LCS runs all the proprietary software that is used to configure the control system. All of them run Windows XP and are members of a windows domain. Users do not have administrative rights on an LCS. All access to LCS’s would be defined by security policy.

The Enterprise Building Local Gateway (EBLG) was created as the gateway and translator. All operational access to the building system is through the EBLG and is exclusively by web services. Maintenance or reconfiguration is performed on the LCS instead.

The Enterprise Building Management System (EBMS) was conceived of as a central harvester of all information from all the web services on all the EBLGs. No software or hardware to work with traditional control protocols would be installed on the EBMS. No traditional controls enter or leave the EBMS.

Using web servers to access the information on the EBMS, operators can see everything in all systems, no matter what the brand or the protocol. This provides a web based point for operating all the buildings. We carefully defined a distinction between Operations and Maintenance/Configuration. You can think of it as the difference between driving a car and performing service on a car.

EBMS – Why we built the UNC Enterprise Building Management System

This article begins a series of articles about a particular large system at UNC

We had a big problem at UNC. Our energy management control room was filled with different so-called enterprise systems. Our buildings were filled with every sort of low bid system that could be purchased under state rules. Where we had been able to impose a standard, internal incompatibilities, between systems from twenty years ago and from ten years ago supplied by the same vendor were as bad as those between product lines. The men in the control room were running a half dozen different incompatible monitoring systems, and all of them were fraught with problems.

As we built out the campus network, it was no longer acceptable to isolate the building systems onto separate webs of leased phone lines. As we moved the systems onto standard networks, the problems became worse. The first attempts by the building systems vendors worked only ion fully isolated networks. Many systems implemented quirky variants of TCP/IP that operated only with certain brands of router. No two chose the same brand of router, and none chose the brand of router selected by out enterprise networking group. With clever trickery, and great expense in time and labor, we were able to work around these problems.

It was surprising how badly the control protocols were written. Developed in isolated labs, by engineers who understood little about the challenges of sharing a network with unknown others, they were really quite bad. Not only were all aspects of these installation discoverable, but they were sensitive. I know one campus whose coal plant control system was taken off line each time an un-configured LaserJet was put on the campus network.

Enterprise control protocols of the day relied on security through obscurity. Anyone who could read BACnet, or LON could discover and operate the entire network. Proprietary protocols such as Metasys or Ultavist were no better. So again, through hard work, and no small expense in network equipment, and even greater expense in time and labor, we put in a complex network of VLANS operating inside the diverse business VLANS of the were able to work around these problems.

The real problems were much worse than I am letting on. Some systems would stop communicating if a router were switched out – until each system in the subnet and neighboring subnets had been simultaneously cold booted. Network stacks written by controls systems houses were abominable and there network interface cards were expensive. I remember tuning network ports down to 10MHz half duplex, to support a $1000 dollar Ethernet interface, while I was buying 100MHz full duplex network cards for $19.

With all the work, with all the effort, with all the off balance sheet costs, I knew the game had to change.

Abstract for Grid-Interop

No time to blog today - I had a deadline for the final draft of my paper for Grid-Interop. Grid-Interop is a conference presented by the GridWise Architectural Council to engage public and private electric systems stakeholders in the integration issues in creating a more efficient, reliable, and resilient electric system.

Long time readers may have noticed how hard I have struggled to lose the academic tone of voice in writing. I hated to see it come back... 

Well,here is the abstract.

True Scalability and interoperability require abstraction and security. Most control systems today expose name/value tag pairs as their interface. Interaction with exposed tag pairs requires a deep understanding of the underlying systems. Secure interaction with sets of tag pairs can only practically be exposed as monolithic yes/no decisions for the entire set. Lack of abstractions, in both process and security, are a barrier to new business interactions.

The smart grid will require integration with smart buildings and their associated power capabilities. Abstract models for system interaction will enable large-scale system integrations. Abstract service models will hide underlying system detail while exposing the diverse systems for orchestration.

Security is the application of policy to service. Situation awareness is required of any mature security model. Situation awareness is only useful when applied to abstractions above identity, above process, and above function. Only when these abstractions are defined, can one then define security.

Service abstractions and Security Abstractions must develop together. Security enables the open provision of business services. Defined services enable the definition of business policies. Service and security together enable open trustworthy interactions with third parties.

If you want find out more, see you in Albuquerque