Products We Almost Bought

Before we settled on our current Enterprise Building Management System at UNC, we came close to buying several off the shelf products that just could not fit us into their marketing model. We were OK with their revenue goals, large as they were, for selling their product to a campus our size. We could have made their architecture work. Their bits were already shrink-wrapped and on CD. Their products would have been a less risky buy. Somehow, they just couldn’t bring themselves to sell it.

We had a few hard and fast goals going in.

1. We wanted to get low level control protocols off the enterprise backbone.
2. We wanted to remove spurious interactions between the embedded systems of disparate buildings.
3. We wanted to be able to orchestrate behaviors across a large campus.
4. Where previously some “less important” buildings had been no cost effective to control, we wanted to extend control to all buildings. It was acceptable to have less nuanced control in simpler buildings.

We also wanted to add some new enterprise interactions that required that we be able to interact with building systems at a higher or more abstract level.

5. We wanted to be able to open up direct access to reading certain kinds of information to our customers [tenants]. This requires that, say, a temperature be recognizable as a temperature and mapped to a particular space.
6. We wanted to be able to expose a programmable interface to functions such as operation scheduling (not configuration) to customers.
7. This required that we have a security model of services that we could map to organizational staff and responsibilities.
8. We wanted to be able to compare live operational postures with live meter readings to determine how effective our energy saving modes actually were.

A few products came close to this. Enterprise Energy Management Systems consumed lower level protocols and offered user interfaces for the control room. They allowed us to define operation postures (code blue energy saving) to invoke when we needed to. They offered Web Services interaction with the enterprise so we could let enterprise developers work with them.

We just couldn’t get those vendors to sell us any product. You see, the market model we proposed didn’t fit the way they were used to selling it.

These applications were all designed with a traditional focus on The Man In The Control Room. They brought all control protocols back to the Server In The Control Room. They exposed a limited number of postures for the entire campus through their web services.

We wanted to use these applications differently. We wanted to place an “Enterprise Server” at each building. All control protocols would then never travel outside the building. The limited number of web services definitions for the campus would then be a plentiful number to use for each building. Security could then be defined for each building, rather than for the whole campus.

The vendors priced their product for one server for campus, and the price was steep. We were willing to pay a premium for our model, but not a multiplier as large as the number of buildings we had. Could we arrange some sort of site licensing, letting the vendor achieve their revenue goals for a project of this size, but letting us install the bits in multiple buildings?

The salesman would nod, frown, and have to go back to headquarters to check. In multiple visits, they would report back that they had been unable to get an answer.

Some of the large controls companies and energy companies have asked me since then why we didn’t buy their product. My only answer is, “We tried. You couldn’t find a way to let us buy it.”

The Argument for Market change: Case Closed.

Friday, Lynne Kiesling blogged about the problems illustrated by the New York blackout last Wednesday (http://www.knowledgeproblem.com/archives/002106.html). I repeated some of my comments about generally bad power and the inability of the current market to address them.

Another poster, Ed Reid replied (I abbreviate, but the link above will let you see the original)

My neighbor, Toby, has different concerns about reliability than the typical regulatory commission, including North Carolina's. Most regulators are concerned about cumulative outages of less than one hour per year (99.98% reliability). The outages he refers to here are mostly either momentary "reclosure events" or local outages resulting from vehicle accidents. I suspect some electric utilities would be interested in selling reliability of the type Toby is concerned about, but I suspect very few customers would be interested in buying it, both because of the extremely high cost and the relatively modest inconvenience associated with these outages.

Ed is clearly knowledgeable about the current grid, but let me re-phrase this.

“The quality you seek is not what the Utilities Commission does – be content with what they have decided is good quality. The current architecture is not really able to give you the power modern electronic systems expect. If we do things the same old way, it will be too expensive to fix. It will only be offered if state commissioners decide there is a market for it.”

I could not possibly have made a better case for market change, for de-regulation, and for the benefits of switching to a market that offers incentives for innovation.

Making Decisions when Everyone Lies

On Wednesday afternoon around 3:45 the power went out in areas of Manhattan and the Bronx in New York City. Although the outage lasted less than one hour, it was sufficient to cause traffic congestion and inconvenience to almost 400,000 customers on a hot, humid summer afternoon.

Or so the official story goes. There was a one hour problem. The problem was gone in an hour. There was some inconvenience. And, as Dorothy Parker famously concluded “…And I am Marie of Roumania

In the 70’s, the American economy suffered through structural problems caused by more than a decade of high inflation. The effects of this inflation were a number on the nightly news. If it were a slow news week, a grocery manager and a housewife would be interviewed, with some statement that consumers were switching from steaks to ground beef. Well, I grew up as one of 11 children – we always ate a lot of ground beef.

The official line was there is a small problem. We can simply quantify it. There are some small concessions that must be made at the grocery store. All fine until the house of cards fell, and all the economic decisions based upon continued inflation compounded, and all the financial games to beat the system by sneaking around the facts came together in a no-job economy with 20% mortgages…

An essay read as I completed High School, one that I cannot find now, struck me. It was so clear, and predicted so much, that I will try to recreate its theme:

“Living with inflation is living in a country where everyone lies about all the most important facts. Every time money is borrowed, both parties are lying about what it will cost to pay it back. Every time someone sells an asset, every home, every factory, every business, they lie to themselves about whether they made money Every investment is made not to increase productivity, not to find value, but to game the lies that everyone tells”

Welcome to WIN buttons and cardigans, and to gas lines and the misery index.

Today we lie about power. We lie about its quality and availability. We lie about the cost of outages. Reliability cannot be purchased on normal markets at any price, because it’s not for sale. We lie about the long term costs of power incidents, and the long term destruction of assets and their resulting accelerated depreciation.

Friday I was checking up on the dorm opening date to drop my daughter off at NYU. The University had an alert on the front page:

Thursday, June 28, 2007, 5:30 pm
Power has been restored to the Fairchild Building. Based on our conversations with Con Edison, we expect normal power operations at the building tomorrow. All employees should report as usual, and operations and activities should go forward as scheduled.

Hmmm – so a one hour outage took the building off line for the entire next day. Another lie about the costs of the outage.

Such outages also cause long-term damage to capital assets. Across Manhattan, Compressors on air conditioners and refrigerators will fail early because of the outage, causing unmeasured expense and loss of service to office, restaurants, real estate owners, and residents. Water systems in high rises already suffered one pumping failure, and will again when the motor fails early, causing turbidity, and cases of minor illness. These illnesses, like the failures, will occur months from now and never be associated with this outage.

We have the most fabulous information system ever devised; the flows of money in an open economy. When piles of cash start getting clogged up in portions of the economy, innovators compete to get it flowing freely again. When we tolerate a closed economy, we create poor information and poor allocation of resources. We externalize costs in ways that we do not recognize. We make bad decisions.

We have a closed regulated energy market wherein the true costs of poor quality power, and even how poor that quality is, is hidden behind opaque markets run by utilities commissions. We do not know the true costs of the energy supply, in particular, how it varies with scarcity across the day. And because the energy market sits just outside our vision, invisible and uncontrollable, we let ourselves believe that it is of good quality.

If we knew the costs of power throughout the day, we would demand knowledge of its quality as well. If the quality were to become visible, we would begin to recognize the correlations, not only of big events but also of small events to PC failures, and to window AC unit failures a month after a poor quality power incident.

If we knew the costs, we would wonder why it isn’t cheaper to fix it. If we knew the true size of the costs, the fixes would not seem so expensive. If we were willing to buy, the innovations that will make real differences in power consumption, in power reliability, in cost of living, and even in carbon use would come to market.

But we have chosen closed markets in power. We have chosen markets wherein the important information is hidden from the consumer. We have chosen markets without instant access to true pricing and true quality. We have chosen markets based on half truths. And half truths are lies.

And because we lie, we make poor investment decisions, and it appears to cost too much to fix.

Diversity in Grease

Did you see the headlines this weekend? Syntroleum is near producing the latest biofuel – diesel from Chicken Fat. I have no idea how much diesel they can get from chicken grease. I can imagine a fuel refinery near each poultry, hog, and beef processing plant. It just makes sense. In particular, it makes sense that it is not THE ONE WAY. Few are likely to advocate larger herds of cattle for more rendering.

Going forward, biogrease will be one fuel among many, and this will stabilize the economy. Cars will be adapted to handle a range of fuels, and people will use what works and is priced right. No one will want a car which will burn biodiesel but will require you, in advance, to pick only dirty French fries, or only poultry, or only beef.

Building controls are still marketed with a choose one source one time model. This control brand, this control protocol is THE ONE WAY. All sales cycles are slow and expensive because the customer must chose THE ONE WAY. Customers are reluctant to choose, because the compromises in the future are unpredictable. Once someone has picked one kind, one time, they specify that forever.

In some ways, it like cars in the 60’s and 70’s. All of the American car manufacturers were focused on getting that lifetime customer. They spent a lot of time on life time image, but not very much time on performance, or quality, or price. Now those companies are struggling in and out of bankruptcy. Is there an end in sight?

Or maybe buying building systems is like computing in the 70’s. Pick a minicomputer brand. You will have to buy big computers from them, small computers from them, networking from them, printers from them…I know, I was an OEM for one of them, I lived and breathed its product line, and sales went away when my customers discovered choice.

Today our problems are bigger. TO meet the energy challenges of the next years, to build zero carbon buildings, we must have top flight competition on quality and performance of systems. The controls, well, they will come direct from Asia and get cheaper and cheaper. Systems will matter.

We can no longer accept systems that are specified by vendor and designed by the low bid contractor standing on a mud bucket. We need systems that are actually designed and engineered for performance. Those who are good will be specialists, and they will be solar engineers, or HVAC engineers, or Storage engineers. We will demand they warrant their work. They will demand that they not be blamed for someone else’s.

In short, building systems will be smaller than they are today. They will be more diverse. Designers will be able to mix and match from best of breed systems, and not settle for one large system designed by a control protocol, be it LON or BACnet, or Tridium, or Metasys or Knx. Those standard will compete within control systems, but they will not, cannot be important at the enterprise level. Eventually they will not be important at the building level. Only at the system level.

And those who do not adapt, will, like the minicomputer makers, be little more than spots of grease.