Unveiling the Unseen World at UNC

Many problems of sustainability stem from costs that are invisible or are ignored. Building performance is rarely managed as building control systems are rarely accessible. As long as building systems are invisible and uncontrollable, then they will only rarely be operated with more than minimal efficiency.

We are just completing a multi-year project at UNC in exposing building control systems using ecommerce protocols. We called this system the Enterprise Building Management System (EBMS).

We had hundreds of existing buildings, each with its own low-bidder installed systems.

Most solve such complexity by opting for sole source acquisitions. Some areas, however, have special purposes that demand a special solution that may not be available from a sole source. In any case, we were starting with a substantial installed base. Merely achieving a traditional sole source solution applied to hundreds of buildings would take decades to complete, and cost far more than our budget.

We placed each system securely in its own sandbox, keeping the low level protocols off our network while exposing web services interfaces to operations staff and our customers. Our central monitoring and operations, based upon open source software, can now interact with systems from each vendor. Its interface is exclusively the web browser, and we have used it, and its graphics, on devices ranging from PCs to iPhones. We are planning to introduce these interfaces to the wider campus, faculty, and students.

While we were planning EBMS, I recognized we needed a new model for interactions with building systems, to open them up to the interaction with business operations, and make their operation visible to building occupants, to students, and to the public. Low level standard protocols such as BACnet and LON and proprietary vendor protocols require deep integrations while shielding information from the non-elect. Even when based on low level standards the supervisory systems are proprietary and without programming interfaces.

Five years ago we decided to move oBIX to OASIS for standards development. We lost several members of the team at the time, members who wanted to develop the standard within the traditional building system community. Web services are the protocols used for business-to-business interactions, but not all web services are usable by business or e-commerce. OBIX uses ecommerce style interfaces so that business programmers with normal skill-sets can interact with buildings.

The approach opens up systems to market competition and innovation. Each building system can be chosen based upon performance and functional fit, rather than compatibility to installed base. We anticipate this will help us see more rapid improvements in the future.

With this work in place, we are now moving forward to more exciting opportunities that our open interfaces create. We are working with the Registrar to schedule building operations around actual use, exchanging information using the same protocols used to schedule meetings by email. This will enable us to schedule heating cooling tied to actual building use. We are opening up environmental monitoring information to building users. The data center can see the building’s operating posture. Archeology can see each collection's environmental conditions.

EBMS – Slogging through the Details

This post continues a series about a particular large system at UNC

The first hurdle we had to solve in our EBMS project was containing all building controls protocol traffic and keeping it off the enterprise backbone. We used a variety of approaches for this, as we had considerable diversity in our installed base. Some systems ad compatible purpose-based hardware that we could plug right in. For others, embedded Linux systems translating older protocols to Web Services were available. For still others, we had to craft custom integrations running on top of a Windows PCs. We carefully defined a distinction between Operations and Maintenance/Configuration. You can think of it as the difference between driving a car and performing service on a car.

Under EBMS, we operate and monitor all buildings using exclusively Web Services. This transition enabled us to get our first buy-in from the enterprise IT group. The network demands and fragility of traditional control protocols caused them many problems. We simplified their life by using only well understood and well behaved web services.

We called the gateway between intra-building control interactions and extra-building monitoring and operations the Enterprise Building Local Gateway (EBLG). We had to accept some diversity of EBLG to encapsulate the much greater range of underlying systems, protocols, and even product cycles.

The most obsolete systems were fronted by oBIX surfaced JACES from Tridium

Standard vendor web services were used for many systems, including iLONs and TAC Soap Services.

Custom mappings of many systems were created using GridLogix software atop Windows XP.

Configuration and maintenance remains un-standardized. We installed Local Control Stations (LCS) for many of the buildings. Each LCS is a locked-down domain-secured PC running Windows XP. The proprietary applications for configuration and control for each building were installed on the local LCS. Some effort was required to make these applications work because of the outdated programming and security models that are rife in the building controls industry. These efforts paid off in that we were able to fully secure these systems for the campus network and to use managed patch application to keep them secure. Control system technicians are authorized to access these systems by remote desktop based upon their organizational position and status.

These are the key components of EBMS. We allow only standard well-known, well-behaved protocols on the campus backbone. We reduced the number of building interfaces to as few variants of web services as we could. All PC-based systems in EBMS are centrally managed, with policy-based patch management and security control. All systems allow remote access using directory enabled security. Technicians can get to the systems they support remotely using the account and password they use to perform the rest of their work.

EBMS – Bringing the EBMS Project to life

This post continues a series about a particular large system at UNC

It is said, you go to war with the Army you have, not the Army you want. And you go to bid with the contractors you have, not the contractors you want. And you specify the technologies the market has, not the technologies you want. And we had to buy the Enterprise Building Management System (EBMS) we could rather than the one we wanted.

When we were finalizing the specification for the EBMS, Web Services was all the buzz among control vendors. Many vendors demonstrated SOAP-based building gateways to us – but the product never quite had a part number. Standards were non-existent. oBIX was not yet even a committee draft. The BACnet-WS committee had not begun to meet.

Administrative IT at Universities is always behind the times. The culture that cherishes and maintains Old East, the first State University building, preserves and protects old business processes as well. Our CIO had pronounced a year before that he was pretty sure that XML was a fad, and he knew Web Services were a fad, and we were to let none of that stuff in here. We needed innovation; we needed to push the building controls industry. Such efforts require very clear communication. We went to bid with a document that never once mentioned Service Oriented Architecture, or Web Services.

Building system contracts are traditionally managed as construction projects. UNC is a state agency. State construction projects rely on a very structured business process producing very predictable results. Because we were breaking new ground, we needed to contract for performance, not design. Performance contracting is not part of the normal process in the North Carolina State Construction Office.

State construction did not know what to do with this project. Our creative and resourceful energy manager, Jay Evans, continued to fight for EBMS. At one point, we actually received a suggestion to bid a small portion of the project and do most of the work on change orders; Jay rejected that approach and kept pushing. Finally, State Construction agreed that we could run a controls project through the State IT business process; SIPS supported an RFP / performance based project cycle.

So it began. All communications outside the building were going to be XML based, but no standard was specified. All Monitoring and Operation was going to be from a single web-based system. All graphics were going to be vector based, so every page could display on any device, and scale as needed from large screen to small. (And this worked – I’ve already seen our system operated from an iPhone).

Fortunately, every group associated with the final team approached the project as partners. An example of this occurred about six months in. To preserve legacy systems until the next building renovation, we were placing special purpose gateways in front of the existing building systems. The subcontractor installing these gateways, Yamas, was using the only solution available at bid time: a proof of concept web services engine available from Tridium. oBIX had become a committee draft, and was sliding into the slow process of OASIS Committee Standard approval. Based on a Phone call, Tridium agreed to re-write their gateways around oBIX and deliver us the newer product based on the standard. Yamas agreed to accept later delivery and still meet their delivery schedule to provide us a standards-based solution. Cyrus Technologies promptly agreed to revamp their development work our standards-based vision. All willingly cooperated as a team to deliver the innovative standards based solution we wanted.

One last system hurdle rose up in our path – standards-based XML graphics. When we launched the project, SVG (Scalable Vector Graphics) – a vector-based W3 standard - seemed the clear choice. During the project, the market leader in SVG tools and rendering bought a market leader in a competing technology; Adobe abandoned SVG after it acquired Flash. We had to scramble and switch to XAML. I am not entirely happy with that change, but that was the way the market developed. On the bright side, there are some DWG/XAML converters, so it looks like we will be able to bring our AutoCAD floor plans into EBMS.

Open Standards-based development is new to building systems. There will be hurdles that you do not expect. You need a team that is willing to commit to a vision. You will probably need to define a performance based project, as the road is not entirely clear. For us, the results are well worth it.

EBMS – Design Decisions for the UNC Enterprise Building Management System

This post continues a series about a particular large system at UNC

As described in the last post, Building System protocols suffer from a number of problems. They are unstable. The protocols are not standards compliant. They have poor security characteristics. So we decided to put them in a sandbox.

Each buildings system was to be isolated within that building. No protocols not easily recognized by Network Support Technicians would be allowed on the backbone; BACnet, LON, et al., were all gone, even their IP versions. All network traffic for control systems would come from protocol stacks that were written for mainstream use. Systems would be secured using our standard network security

All connections to building systems be sessionless. This one needs some explanation. When a program on your computer connects to a database, you log in, and establish a session. That connection is maintained by regular hand shaking. The remote server remembers things about you, so you can continue the conversation. This is efficient, but it does not scale well.

When you contact a web server, you connect, get your page, and then the web server forgets all about you. This allows one web server to service thousands of clients. The standard for sessionless program to program communications is web services. You use web services whenever Gmail types ahead to finish an address for you. We opted to use web services for all routine communication.

We defined two ways to connect to the sand box. It was necessary that we choose names that were not tied to any particular brand, so we made up our own. All communications between the EBMS and the control system are through the Enterprise Building Local Gateway (EBLG). All maintenance of the building system is through the Local Control Station (LCS). Both access points are secured by network security.

We defined a Local Control Station (LCS) to support the needs of each individual system. The LCS runs all the proprietary software that is used to configure the control system. All of them run Windows XP and are members of a windows domain. Users do not have administrative rights on an LCS. All access to LCS’s would be defined by security policy.

The Enterprise Building Local Gateway (EBLG) was created as the gateway and translator. All operational access to the building system is through the EBLG and is exclusively by web services. Maintenance or reconfiguration is performed on the LCS instead.

The Enterprise Building Management System (EBMS) was conceived of as a central harvester of all information from all the web services on all the EBLGs. No software or hardware to work with traditional control protocols would be installed on the EBMS. No traditional controls enter or leave the EBMS.

Using web servers to access the information on the EBMS, operators can see everything in all systems, no matter what the brand or the protocol. This provides a web based point for operating all the buildings. We carefully defined a distinction between Operations and Maintenance/Configuration. You can think of it as the difference between driving a car and performing service on a car.