MathJax

Friday, November 10, 2017

Paying People Not To Use Electricity - ACCC Submission

This is a response to calls for feedback on the ACCC Retail Electricity Pricing Inquiry - Preliminary Report[1]. This response is intended to encourage the ACCC to look more closely at perverse forms of demand response. Background articles in this series are Paying People Not To Use Electricity - A Beautiful ScamPaying People Not To Use Electricity - The Fatal Flaw and Paying People Not To Use Electricity - The Economics.

Thanks to colleagues at MakeHackVoid who provided advice on this submission, in particular Paul Harvey and René.

ACCC Submission


The ACCC Retail Electricity Pricing Inquiry - Preliminary Report [1] concludes demand response "has the potential to facilitate consumers reducing their demand at peak times, and could thereby reduce the need for costly new generation and network infrastructure."[2]. Demand response though, takes many forms and the inquiry has thus far not considered a form of demand response that contains perverse incentives and which involves payment by volume for a volume that can't be measured. The preliminary report states the "ACCC will be mindful of the history of interventions in this market which have too often had unintended consequences to the detriment of electricity users." It is likely that the intervention in the  wholesale market to create mechanisms to provide payments to consumers and non consumption aggregators for not consuming electricity is  another of those interventions. The concept has got a lot of attention in the popular press[3][4].

While not described this way by proponents the gist is to provide electricity to consumers at below cost and then pay consumers not to buy it. It is impossible to measure electricity not consumed so non consumption must be estimated from historical usage. This provides an incentive for consumers on fixed rate supply contracts to shift loads as far as practicable into times of high demand to maximise the payment for switching off, the opposite of what is desired. Air conditioners in particular could be more profitably run flat out whenever wholesale electricity prices are high as it is hot weather that causes wholesale price peaks and they provide a large load that can be switched off for payment. The argument for selling electricity at below cost is contained in a 2015 CSIRO study[5]. It states,  "Consumers are particularly resistant to real-time pricing and (especially) capacity pricing, presumably on account of their greater novelty and complexity (hence, perceived risk), and pervasive mistrust and rejection of the concept that electricity should cost more depending upon demand."[6] Therefore a "flat rate tariff offer with money-back guarantee achieves an unparalleled level of consumer acceptance, unmatched by any other combination of tariff and risk relief."[7] Hardly surprising really, who wouldn't want to consume what they like at below cost and get paid if they don't.

The rational alternative for demand response is for retailers to sell electricity to consumers at cost plus a margin. This has the advantages that:-
  • The amount consumed can be measured. 
  • Consumers who choose to switch off can capture the full benefit of doing so rather than having to share it with a service provider. 
  • Consumers can still seek assistance from a service provider when the effort of monitoring prices plus managing loads is too high for the benefit gained without assistance.
  • The financial incentive is to shift consumption as much as practicable to times of low demand which aligns with the goal of reducing peaks in demand.
There is also a push to take this further and implement a scheme where non consumption of electricity can be used to increase prices for consumption of electricity by selling non consumption into the electricity market as if it was equivalent to electricity generation. The rationale for this is described in a ClimateWorks report[8] where selling non consumption as if it were production is described as necessary because reducing consumption "has the potential to reduce market prices at peak times when the marginal generator is high cost. However the benefits of the reduced price are shared by all market participants and cannot be effectively captured by the DSR provider"[9]. Therefore, non consumption should be able to be bid into the market as equivalent to generation which has the effect of increasing the wholesale electricity price.

There is an example of this being trialed currently. As reported in the Financial Review[10], "Intercast & Forge have committed to deliver 10 megawatts of electricity off the grid when asked to by the Australian Energy Market Operator, for which it will receive $323,654 in funding from the Australian Renewable Energy Agency"[10]. The "company had already saved $600,000 on their power bills in the past three months alone - from being on the spot market rather than a long-term contract with a retailer as well as turning their four furnaces off for a total of 39 minutes, normally for five minutes or less, during peak periods."[10]

So this large consumer is already reducing demand during peaks by responding to price signals for their own economic benefit but is now to be paid extra to maintain consumption until asked to switch off. If that payment is to come from other electricity consumers through an increase in the wholesale price, as eventually intended, it can only result in a sub optimal economic outcome. It creates an obligation to maintain consumption when it is already uneconomic until asked to switch off, and an incentive to consume when uneconomic so as to have consumption that can be sold as available to be switched off. Funds extracted from other consumers through higher prices will be used to fund uneconomic consumption, to the detriment of other electricity users.

An attempt to introduce a demand response mechanism into the wholesale market was rejected in 2016. According to Finkel the "AEMC decided not to introduce the proposed mechanism on the basis that it would be costly to implement and that consumers can already contract with retailers and specialist providers, and can choose to be exposed to the wholesale market spot price through their retail contract"[11]. The reasons given by the Australian Energy Markets Commission for rejecting this rule change request in the Final Determination[12] under the headings "Demand response mechanism" and "Overview of determination to not implement the DRM" remain valid and should be considered by the ACCC in the interest of consumers. Finkel goes on to say "If unscheduled participation in the wholesale market as proposed in the 2015 rule change is not appropriate, there are other options in use around the world, including demand response participation in reliability markets in New York and Texas. The important thing is that a suitable option capable of unlocking the vital benefits of demand response is chosen"[11].  The design of the NEM means the "suitable option capable of unlocking the vital benefits" is the exposure of consumers to the wholesale price and despite the claims of proponents "there are no DRM –like arrangements in any market that is designed similarly to the NEM"[13]. Eastern Australia has a wholesale price, which can be 150 times the average price during peaks and many consumers, including Intercast & Forge[10] are reducing consumption when the price of electricity is too high. Electricity plans that made wholesale plus a margin rates available to smaller consumers would make that opportunity available to all and encourage manufacturers to incorporate automation in devices that balanced desirability of device consumption against current price.

There is a variety of mechanisms suggested for paying people not to consume electricity but they all suffer the same fundamental flaw. The amount of electricity that would have been consumed in the absence of payment not to consume it is unknowable. Under a likely model, aggregators will be selling non consumption by volume into the wholesale electricity market when the volume can't be measured. In the interests of shareholders they will be obliged to sell as much as possible, so the aggregation business is likely to become another of those interventions with unintended consequences to the detriment of electricity users.

Further Reading


Some background material was also written in preparation for this submission which is available at blog.urremote.com

References

  1. Australian Competition and Consumer Commission (2017), "Retail Electricity Pricing Inquiry Preliminary report", 22 September 2017, https://www.accc.gov.au/system/files/ACCC%20Retail%20Electricity%20Pricing%20Inquiry%20-%20Preliminary%20Report%20-%2022%20September%202017.pdf
  2. ibid, pg 86
  3. Charis Chang (2017), Households will be paid to reduce power consumption during peak periods, News.com.au, http://www.news.com.au/finance/money/budgeting/households-will-be-paid-to-reduce-power-consumption-during-peak-periods/news-story/5ce9fc72bdf789569d8069fdbdf45e39
  4. Sheradyn Holderhead (2017), ARENA projects to prevent blackouts deliver 200MW of capacity to the grid, October 11 2017, TheAdvertiser, http://www.adelaidenow.com.au/news/south-australia/arena-projects-to-prevent-blackouts-deliver-200mw-of-capacity-to-the-grid/news-story/c9c3c7932d1d8797076d423d83786d09
  5. Karen Stenner, Elisha Frederiks, Elizabeth V. Hobman and Sarah Meikle (2015), "Australian Consumers’ Likely Response to Cost Reflective Electricity Pricing", CSIRO, Australia, https://publications.csiro.au/rpr/download?pid=csiro:EP152667&dsid=DS2
  6. ibid, pg 26
  7. ibid, pg 6
  8. ClimateWorks Australia (2013), "Industrial demand side response potential Technical potential and factors influencing uptake Initial findings and discussion paper",ClimateWorks Australia,  February 2014, https://climateworks.com.au/sites/default/files/documents/publications/climateworks_industrial_demand_side_response_potential_feb2014.pdf
  9. ibid, pg 16
  10. Mark Ludlow, Ben Potter, Angela Macdonald-Smith (2017), "Powershop, United Energy, metal foundry sign up for demand response trial ,Australian Financial Review, Oct 11 2017, http://www.afr.com/business/energy/electricity/powershop-united-energy-metal-foundry-sign-up-for-demand-response-trial-20171010-gyydxu
  11. Dr Alan Finkel (2017), "Independent Review into the Future Security of the National Electricity Market: Blueprint for the Future, The Commonwealth of Australia, pg 148, https://www.environment.gov.au/system/files/resources/1d6b0464-6162-4223-ac08-3395a6b1c7fa/files/electricity-market-review-final-report.pdf
  12. AEMC (2016), " (Demand Response Mechanism and Ancillary Services Unbundling), Final Rule Determination, 24 November 2016, Sydney, http://www.aemc.gov.au/getattachment/68cb8114-113d-4d96-91dc-5cb4b0f9e0ae/Final-determination.aspx
  13. ibid, pg 18

Friday, November 3, 2017

Paying People Not To Use Electricity - The Economics

In the first article in the series on paying people not to consume electricity I identified the idea as a beautiful scam and in the second explored the fatal flaw. Here I examine the economics of paying people not to consume electricity.

The Economics Of Paying People To Switch Off


Retailers buy electricity in the wholesale market at a price that is set every thirty minutes and sell this electricity to customers at a fixed price that reflects an average wholesale price plus a margin (see IPART Report). Consumers are paying of the order of $0.20 per kilowatt hour (exc. GST ACTEWAGL ACT November 2017) but at times of extreme demand the retailer will be paying the maximum market price of $13.80 for that same kilowatt hour (January 2015). Therefore there is $13.60  saved if the consumer doesn't consume that kilowatt hour of electricity, a portion of which can be payed to the consumer to encourage that outcome. So the funds for paying people to switch off comes from the savings made by not selling them electricity at below cost in the first place.

It is impossible to know how much electricity hasn't been consumed unlike the emperor who had to suffer the  ignominy of overwhelming evidence.


The economics is made more complicated by it being impossible to know how much electricity would have been consumed in the absence of a payment. Therefore, an amount of electricity not consumed does not necessarily equal a reduction in electricity generated. It is hardly surprising then, that when AEMO ran a trial "the funding round had well exceeded the 160 MW initially hoped for, and cost less than expected". Ultimately there is no limit to the amount of electricity that isn't consumed.

Rational Economics


An alternative to selling electricity at below cost then paying consumers not to buy it is to sell it at cost plus a margin. This has the advantages that the electricity consumed can be metered and the price signals encourage consumers to use less when the price is high. It's also efficient, the consumer captures the whole economic value by paying nothing for what wasn't consumed.

Why Not Rational Economics? - One Reason


To quote from a 2015 CSIRO study titled "Australian Consumers’ Likely Response to Cost Reflective Electricity Pricing" demand response is not being structured this way because  "Consumers are particularly resistant to real-time pricing and (especially) capacity pricing, presumably on account of their greater novelty and complexity (hence, perceived risk), and pervasive mistrust and rejection of the concept that electricity should cost more depending upon demand." Therefore a "flat rate tariff offer with money-back guarantee achieves an unparalleled level of consumer acceptance, unmatched by any other combination of tariff and risk relief."

That is the same argument that was advanced when I was part of this team back in 2004. So there you have it, consumers "rejection of the concept that electricity should cost more depending upon demand", is the intellectual justification for paying people not to consume below cost electricity. Hardly surprising really, who wouldn't want to consume what they like at below cost and get paid if they don't.

Why Not Rational Economics? - Another Unspoken Reason


When I was working on this electricity was cheap reliable and intervals of extreme pricing were rare. Genuine improvements were hard. I estimated there was very roughly $100 per year of value available for a residential consumer with an air conditioner, who avoided price peaks. Capturing this value required new interval metering, new electricity plans, information systems to convey price data automatically to devices and devices able to respond to price signals. This is challenging to do for less than $100 per year and framing the problem this way makes it obvious. If you can frame demand response as people getting paid for "reducing the need for supply-side infrastructure" which "delivers lower electricity prices to all consumers" and also add some mystery it is far more salable (Reducing electricity costs through Demand Response in the National Electricity Market A report funded by EnerNOC).

The product companies could sell is non consumption of electricity as if it was equivalent to generated electricity. It is an excellent boondogle. In the  emperor story - "I'll send my honest old minister to the weavers," the Emperor decided. 'He'll be the best one to tell me how the material looks, for he's a sensible man and no one does his duty better.' The minister however, fearing for his own position told the weavers - 'Oh, it's beautiful it's enchanting.' The old minister peered through his spectacles. 'Such a pattern, what colors!' I'll be sure to tell the Emperor how delighted I am with it." Similarly the researchers would be more successful in obtaining support and in turn provide credibility for the boondogle. Over time the researchers and the demand response industry intermingled.

Another Trustworthy Official


From our story - The Emperor presently sent another trustworthy official to see how the work progressed and how soon it would be ready... He declared he was delighted with the beautiful colors and the exquisite pattern. To the Emperor he said, "It held me spellbound." The government sent their chief scientist who concluded in Recomendation 6.7 that authorities "recommend a mechanism that facilitates demand response in the wholesale energy market" (Finkel review - Independent Review into the Future Security of the National Electricity Market, Blueprint for the Future, June 2017).

Taking It To The Next Level


If there is success in reducing consumption it "has the potential to reduce market prices at peak times when the marginal generator is high cost. However the benefits of the reduced price are shared by all market participants and cannot be effectively captured by the DSR provider." (ClimateWorks Pg 16). So to make sure reducing consumption doesn't reduce price, proponents want to treat reduced consumption as increased generation.

Under current schemes, the savings for paying people not to consume comes from consumers being able to buy electricity at below cost. This limits the people that can be paid not to consume to small retail customers. This innovation provides a method of extending the opportunity to large consumers who don't get electricity below cost. They too, will now have an incentive to switch loads into peaks to raise prices until they too are paid to switch off. The opportunity exists because of the extraordinarily high multiple of peak to average prices of approximately one hundred and fifty.

Depending on the algorithm used to calculate the consumption estimate, it may even be economic to burn electricity just to create an inventory that can be switched off for a payment.

A Case Study


Intercast & Forge is one of 10 companies which have won tenders to supply up to 200 megawatts of "demand response" electricity to help keep the lights on in the eastern states of Australia this summer.

Intercast & Forge have committed to deliver 10 megawatts of electricity off the grid when asked to by the Australian Energy Market Operator, for which it will receive $323,654 in funding from the Australian Renewable Energy Agency. (AFR Oct 11 2017)

The "company had already saved $600,000 on their power bills in the past three months alone - from being on the spot market rather than a long-term contract with a retailer as well as turning their four furnaces off for a total of 39 minutes, normally for five minutes or less, during peak periods." (AFR Oct 11 2017)

So here we have a large consumer that is already reducing demand during peaks by responding rationally to price signals for their own economic benefit but is now to be paid extra to maintain consumption until asked to switch off. The obligation to maintain consumption when it is already uneconomic, until asked to switch off, can only produce higher prices for other electricity consumers and creates an incentive to increase the amount available to be switched off.

Thursday, October 26, 2017

Paying People Not To Use Electricity - The Fatal Flaw

In the first article of this series I identified a beautiful scam. In this article I explore the fatal flaw that makes it possible. It wasn't obvious to me at first and given the community enthusiasm for paying people not to use electricity, it mustn't be obvious to most others either.

Without hearing the proponents pitch, people I've discussed this with have said that's daft I don't believe anyone would think otherwise, just as the implausibility made the emperors story unappealing when I was a child. However, my colleagues thought paying people not to use electricity was inspired. For me the concept is so obviously daft it defies belief that it could have lasted this long and grown into government policy

Once you know, it can never be seen the same way again. The genius can be admired but the wonderment is gone.


It led to this strange situation where I was trying to convince colleagues of what seemed an obvious truth and utterly failing. In retrospect I think, like Shawn Spicer's claim on the inauguration crowd, the greater truth was that if paying people not to use electricity was truly daft the personal consequences would be negative. It's better to prefer alternative facts. The other possibility is I was wrong and there is no fatal flaw. I've thought hard in the intervening thirteen years and I've not been able to devise nor have I heard a good counterargument. If there is no fatal flaw it would be painfully cathartic for me to learn the truth. 

The Fatal Flaw 


If you don't accept the fatal flaw as truth then everything in this series is waffle or perhaps there are multiple truths in accordance with epistemic relativism.

Everything hangs on this:-

  • Contention: The amount of electricity that would have been consumed in the absence of payment not to consume it is unknowable.
  • Corollary 1: The volume of electricity that would have been consumed if people hadn't been paid not to consume it can only be guessed/estimated. 
  • Corollary 2: In any peak some consumers will consume less than estimated for reasons unrelated to being paid.
  • Corollary 3: If you pay by volume for something that can't be measured people will sell you a lot of it.
  • Corollary 4: A consumer not paying market price should consume as high a proportion as possible of their total energy at peak times so as to maximise the amount they will receive when responding to a request to switch off.
  • Corollary 5: A consumer that can increase market price by not consuming should consume as high a proportion as possible at peak times while ever the increase in consumption cost is exceeded by payments for switching off. 

It is as simple as that but a lot flows from it. The whole purpose of demand management is to reduce consumption when electricity is expensive or more often shift consumption from times of high demand to times of low demand. However, the incentive scheme incentivises exactly the opposite until a payment is offered. As well, some would have switched off anyway, perhaps they were out that day.

So paying people not to use electricity means accepting their claims of how much they should be paid or at best guessing while incentivising behaviour that makes the problem you are trying to solve worse. All this when incentives could be aligned with desired behaviour by selling electricity at cost plus a retail margin.

Sunday, October 15, 2017

Paying People Not To Use Electricity - A Beautiful Scam

Paying people not to use electricity is not on the scale of the WMD claim that justified invading Iraq but more exciting for me because I was there when the idea was born, just after that war. It started small and I expected it to die young but it has grown in stature to the point that it gets a mention by Sarah Hanson-Young on QandA, praise everywhere you look and there is now a demand response trial. The basic idea is to sell electricity below cost and then pay people not to buy it. Now that it's public policy, there's an option for the man in the street to climb on the bandwagon and win some pocket money extracted from the pockets of those that ignore the opportunity. Those that take advantage will be paid to help the wider community understand the fatal flaw, but anyone interested should do so fast as surely this one can't last long.

We are a nation blessed with an abundance of sun, wind, coal and gas and back in 2004 this was reflected in Australia having close to the worlds cheapest electricity. That was when a CSIRO team started work on the idea of paying people not to consume electricity. I was part of that team and for a short while shared in the excitement. Nobody is penalised or coerced into contributing to reductions in peak electricity demand but those that make the effort benefit. People can generate value by not doing something and get paid. What could be better than that? After a light bulb moment though, I could never view the idea this way again, it was so obviously daft. I pointed out the fatal flaw to colleagues and suggested we should try other approaches to demand side response. Other approaches though were boring, incrementalist and difficult to make cost effective at the low power prices of the time. Back then, I'd never have predicted that weaknesses in electricity industry oversight would be exploited so successfully that Australia now has close to the worlds most expensive electricity.

Growth in electricity prices over 10 years. Source: Rod Sims - National Press Club Speech, 20 September 2017. For a more complete analysis see a comparison of Australian and US electricity prices over time.

My colleagues were more prescient, the fashion for developing novel methods of extracting funds from electricity consumers was at its dawn. Electricity had been a boring industry for many years until that time. My insights got some support from my boss, who'd also been drafted into the project, but outright hostility from those in charge. They wouldn't confront the fundamental problem, and eventually I was no longer allocated to the project. I couldn't understand the hostility, I thought I was helping the team by suggesting we adjust our goals to avoid inevitable failure. In retrospect I've come to the view my colleagues were far wiser, they were pursuing a more useful truth.

A truth can be harnessed to achieve useful outcomes. A truth doesn't have to be absolute, true in particular circumstances is sufficient, even if that truth becomes untrue outside the relevant domain. We may scoff today, but Ptolemy's geocentric model for planetary motion was used to prepare astrological and astronomical charts for over 1500 years. No one scoffs at Newtons law of universal gravitation. It is a useful truth with continuing practical application but in Einstein's general relativity, gravitational force is a fictitious force arising from the curvature of spacetime. The most favourable outcome a research team can hope for in CSIRO is to be well funded and that requires exciting novel ideas. Boring incrementalist approaches will not be funded, researchers must promise something grander than anyone else. This is a fundamental truth, but everyone's proposals are grand so it's also impossibly hard. When there are many excellent weavers how does a weaver win the contract to supply clothes for the emperor. He must promise something others can not, the finest, best suit of clothes from a fabric invisible to anyone who is either unfit for his position or hopelessly stupid should do it. As a child I hated this story because I thought it ridiculous. I couldn't understand the desirability of keeping up the pretense. As I age, I find the truth it reveals to be profound. In the circumstances of the courtiers it truly was a beautiful suit, the quality of which contributed to their good standing with the emperor and continued well being. There can be no greater success in CSIRO than a world leading project that has managed to grow for thirteen years and become a major public policy initiative. It is a truly excellent project.

An emperors love of beautiful weaving provides for a great yarn which is often interpreted in different ways. A short and seemingly simple tale that's really quite deep.

The emperor looked a bit of a dill after the boy revealed a greater truth but he carried on with the parade. The weavers kept their fee and the emperor kept his throne, so no harm was done. Similarly when Hans Blix was urging caution, the best insider response was to ignore the doubts and discredit him. Those urging war would get to enjoy the spoils, so the existence of WMD was a useful truth even if it would eventually be mocked, much like the fate that befell Ptolemy's geocentric model. To maximise income from reducing electricity consumption in times of peak demand consumers must first shift as much of their consumption as possible to times of peak demand. This is the opposite of the purported aim. Paying people not to consume electricity will eventually lose favour but the emperor will carry on and the CSIRO heroes will keep their honors and promotions. Australians will pay a bit more for electricity but in truth, that doesn't really matter much.



Sunday, September 17, 2017

Electric Unicycle Batteries And The Design Flaw That Makes Wheels Dangerous

Figure 1 - An Airwheel X8 in action

Summary

Electric unicycles are mostly safe but an instantaneous loss of power results in rider injury. Most wheels have a design flaw that causes this dangerous situation to occur as they age. A rider is likely to experience a rare and mysterious shut down on a wheel that was for a long time safe.

This study explains the problem and details investigations into two particular models, an unbranded el cheapo and an Airwheel X8. Both these wheels were flawed with the el cheapo failing quickly. It is possible to avoid shutdowns with the X8 by replacing the battery before it gets old and modifying the electronics can completely avoid low voltage shutdowns.

An interesting observation arising from the study was that adding a second battery in parallel should increase range by a factor of about three.

Introduction

Wheels at low speed are generally pretty safe as you can easily step off. The foot platform striking an obstacle is difficult because the riders body continues forward without the wheel underneath. Some quick foot work is required to get the rider's feet back under their centre of gravity but as they can push off the wheel platform it usually ends well. Motor shutdown on the other hand is dangerous and I'm not the first to observe this. I've experienced it four times and it always hurts.

Why Is Motor Shutdown Catastrophic?

Figure 2 -A shutdown looks similar to this
When the motor is driving forward, the riders weight must be forward of the wheel centre so that the torque produced by the rider matches the torque produced from the horizontal ground reaction as shown in Figure 3 .
Figure 3 - Stable forward motion
If that ground reaction is lost the rider will be hanging in space without a force to oppose gravity and will therefore descend face first as shown in figure 4.
Figure 4 - What happens when the motor stops.
It is worse than tripping over while walking as the rider can not push against anything to try and recover or reduce their rate of descent. No horizontal reaction force is possible. An exclusively vertical force through the feet, which is all that is possible, serves only to increase the riders rate of rotation about a centre of gravity which moves further forward of their feet as they approach the ground. Only hands can be used to break the fall. Hands, knees and sometimes the face will hit hard and suffer injury. The only chance is to pull the knees toward the chest faster than the rider is falling and throw the feet forward. I know its been done but I've never pulled it off. I find I'm already on the ground before I've realised what was occurring.

In this video a rider dismounts gracefully from an unpowered wheel by pulling one knee upwards as he is falling, planting his foot in front of his body and relying on forward momentum to lift his centre of mass above his foot. He is undoubtedly a talented rider but his starting point is one of balance rather than hanging in space and he doesn't have to realise during the fall that he needs to do the opposite of what he would normally do when tipping forward, as is required on a powered wheel.
Figure 5 - This minor scrape was from catching the foot plate and stumbling on the dismount. The shutdown injuries were much worse.


My Four Shutdowns

The first was on a wheel only two weeks old that had a cell in the battery pack fail suddenly, dropping the cell voltage below the shutdown voltage. If you buy the cheapest wheel possible, it will almost certainly have low quality batteries as they are a large proportion of the sellers total expense making battery skimping the easiest way to achieve low prices. This is what led to the hover board failures that made the news a while back.
Figure 6 - Low quality batteries can fail without warning and even catch fire. 

Stupidly, I replaced the failed cell with a cell that had protection which tripped out in use and that was the second shutdown. The wheel seller sent me another battery but took the el cheapo wheel back before the battery arrived. Next I went up market with an Airwheel X8 and had 18 months of trouble free wheeling. Oh what joy! I sensed the battery was aging, removed it to test and fitted the generic pack I'd been sent previously. First time I rode with the new battery it lasted well but the ride ended with a shutdown. I put back the aging battery which had tested as worn and had another shutdown early in the ride. This exacerbated the injuries from shutdown three that hadn't yet had time to heal. Shutdowns three and four were a mystery so some investigation was required.

How Are Wheel Batteries Managed?

The batteries are built from lithium 18650 cells connected in series and packaged with a battery management system (BMS) like that shown in figure 7 and listed on Aliexpress with the specifications in the appendix.
Figure 7 - A Battery Management System (BMS) for a wheel is packaged together with the 18650 cells. Note the 4 FET switches in the top left. Three of these are used to switch off power to the wheel when over current or under voltage is detected and the fourth switches off the charging current when the cells are fully charged.

The BMS manages cell balancing, charging and supplies power to the wheel through switches (FETs) that are switched off in the case of over current or under voltage.

On an X8 wheel there are four battery indicator lights. The number lit reduce as the battery discharges until, at the minimum acceptable voltage the wheel beeps a lot, flashes all four lights and goes into tilt back making it hard to ride.

The battery charger is not smart. It supplies a constant voltage up to its maximum current and shows a red light to indicate charging. The light turns green when the current falls low enough to indicate a full charge. The actual charging current is controlled by the BMS, not the charger. I hadn't realised this at first and, when the wheel was disassembled, ended up damaging the original X8 battery and the generic wheel replacement battery by charging through the discharge cable. This bypassed the BMS charge control and caused overcharging.

What Went Wrong?

It is under voltage that caused all my shutdowns and most other people's as well. The primary problem is a design flaw. Sure, we want to protect batteries from under voltage that will wreck them but it shouldn't be by injuring a rider. This flaw lurks always but will not be triggered by most usage scenarios. Normally the wheel goes into tilt back and the rider dismounts before low voltage shutdown occurs, however under certain circumstances shutdown will occur without tilt back. How this can happen is investigated in the following testing.

Method

The batteries were discharged at a low 0.2 amps rate and the voltage periodically measured. This data was overlaid on published test data for the Panasonic NCR18650PF cells, these being the type fitted to the  Airwheel X8. The voltage at which each battery indicator light changed state was measured and the voltage at tilt back and shutdown was measured.

Results

Figure 8 shows battery discharge data for the X8 battery before overcharging, after over charging and for the generic battery after overcharging. It also shows the per cell voltages for shutdown etc.
Figure 8 - Published test data overlaid with testing data of an aging cell X (black), aging cell after overcharging X (grey) and generic cell after overcharging X (blue). The discharge rate for the X tests was 0.2 amps.

The aged X8 battery is shown to be, as suspected, of significantly reduced capacity and was made worse by a single overcharge. The generic battery after one overcharge is shown to be of much lower capacity than the X8 battery after over charging. The shutdown voltage of the generic BMS was 3.56 volts which was higher than the tilt back voltage of 3.39 volts per cell.

Discussion

When the generic battery was fitted, the long first ride suggests the unbranded cells were quite reasonable. However that first ride was, from the outset, doomed to end in pain as the shutdown voltage was above the tilt back voltage. There would never be a warning before shutdown three. The generic pack had a shutdown voltage of 3.56 volts, the X8 pack was 3.22 volts and the AliExpress BMS advertises 2.1 volts which shows BMS specifications vary, so that even a pack of the correct voltage can't be safely fitted on any wheel. They must be matched. Also, the generic cells were ruined by a single overcharge.

Shutdown four is more problematical because the implication is that all X8s and probably most other wheels will become dangerous as the battery ages. The immediate cause of shutdown four is the damage sustained by overcharging, leading, under heavy load, to an instantaneous voltage drop from above tilt back voltage to below shutdown voltage. While this battery was prematurely aged by overcharging, all batteries will eventually suffer similar deterioration and battery capacity is also reduced when cold e.g in winter. In use this deterioration will happen slowly so shutdowns will be rare, occurring at first when climbing a bump late in a ride on a cold day, as the battery approaches tilt back voltage and becoming more common as the battery ages further. A rider is only likely to be aware of a rare and mysterious catastrophic failure on a wheel that was for a long time safe. All wheels with this design will eventually become dangerous.

Increasing Wheel Range

Another observation from Figure 8 is how little of the available battery capacity is actually used in the X8. At a likely occasional peak of 7 amps draw the wheel goes into tilt back at 1.25 AH which is less than half of the total energy available. Adding a second battery in parallel would reduce the current draw by half and avoid tilt back until about 1.8 AH. So changing out a battery when flat would double the range, but running a second battery in parallel will increase range by nearly a factor of three. With a single battery, I've always found range insufficient but two batteries in parallel would equate to about three hours riding time, by which time the rider is likely to be fatigued anyway.

Why Are Wheel Designs Flawed?

A BMS is used in most lithium battery installations for battery protection and the same logic is ideal for nearly all scenarios, excepting wheels. A battery control chip is required for every cell and the chips to do it are abundant. Ideally a wheel wouldn't do low voltage shutdown but that is what the available chips do. I guess wheel manufacturers have just gone with a standard BMS design. What they haven't done, in the X8 at least, is try and work around this limitation in the wheel control logic and they should have.

What Can Manufacturers Do?

The goal is to try to always get tilt back before shutdown and shut down slowly enough for a rider to react and step off. If there is to be a shutdown voltage, setting it lower, at the minimum cell voltage would help as it makes tilt back more likely to occur before shutdown. This would be 2.5 volts for the Panasonic NCR18650PF cells in the X8 rather than the current 3.2 volts. The 2.1 volt cutoff in the AliExpress controller meets this requirement but if 2.1 volts ever occurred in use it would damage the cells. What they really should do is, not allow battery cells to reach minimum cell voltage during riding without shutting down.

Avoiding minimum cell voltage can be done by reducing current sufficiently to maintain another voltage level below the tilt back voltage. For the X8 controller with the shutdown voltage already reduced to 2.5 volts, perhaps 2.9 volts would make a good minimum.  This only has to occur for a second or two as tilt back will already have been triggered. What the rider will experience is a reduction in torque, probably when it is most needed climbing an obstacle, followed by tilt back. This is far better than shutting down. The wheel will go from rigid to spongy but loss of power will be gradual enough for the rider to react and they will still be able to push horizontally against the wheel, making it possible to dismount.

What Can A Rider Do?

A careful rider can stay in the safe zone. Use quality cells and test a new pack before fitting as there are a lot of dodgy lithium batteries for sale. Store the wheel inside in winter and don't stop for too long in freezing conditions.Then monitor the cells as they degrade and replace them early. Replacing a battery prematurely is wasteful but better than suffering a shutdown.  Alternatively a rider could modify the BMS to bypass the FET switches so that the battery remains connected when the FETs switch off or connect the gate to the drain on the FET switches so that they can't switch off. This will eliminate shutdowns but also over current and under voltage protection for the batteries.

For the X8 wheel the battery capacity lights can be used to measure battery aging. From figure 8 it can be seen that, when new, the 4-3 indicator light transition occurs at 0.5 AH of discharge and the 3-2 transition occurs at 1.1 AH of discharge so there should be about the same ride time between
the two transitions. As the battery ages the 4-3 transition happens earlier but the next 3-2 transition remains about the same amount of time after the previous. This is what I'd observed in practice, 4-3 battery indicator transition was happening earlier in a ride as the battery aged and on the overcharged battery the 4-3 transition occurred immediately on commencing riding. Using data from the graph and the knowledge that I hadn't yet had a shutdown with the aging battery, it looks like a ratio of up to five in ride time between the 4-3 and 3-2 transitions is safe. At some point beyond that it becomes dangerous.

Buying Replacement Batteries Is Difficult

Replacement batteries are hard to locate so most people probably don't replace them. That would suggest there are a lot of old wheels around with dangerously aged batteries that will hurt their riders.

I've found only one supplier on Aliexpress. They don't specify which cells you will get which is annoying but at least they offer a low shutdown voltage of 2.75 volts per cell which improves on the 3.21 volts of the original Airwheel X8.

Figure 9 - Wheels can traverse fairly rough terrain


Appendix

Specifications of a wheel battery management system (BMS) listed on Aliexpress.




Saturday, January 16, 2016

How Hot Is The Lounge?

“When you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind” 
― Lord Kelvin

Approaching Christmas it was hot, enough to keep checking the lounge room temperature obsessively. To keep the house cool without air conditioning the windows are opened at night then closed during the heat. Recently it was an extreme 37° outside while inside was a more livable 27°.
These digital thermometers are great but they only tell you the temperature right now.
A couple of digital thermometers at less than A$2 per piece are used to know when it's hotter out than in and then I have to to keep observing so as to close or open the windows appropriately.

Obsession though, requires better than spot observations.  How cool does it stay inside compared to out, how often is it uncomfortably hot, how cool does it need to be at night to cool the house sufficiently and so on? The house has a lot of thermal mass so how much less effective is this technique when there is successive hot days? Answering these questions requires more data. No problem, I'll get a sensor that records observations and to make analysis easy it should publish the data online.
Wireless Temperature Logger - Front
This sensor from Wireless Loggers looks good and runs on a coin battery but is a hefty A$ 77 per unit.
Ethernet Tag Manager - Front View
These sensors also need a wireless ethernet base station which adds complexity and another A$164 in expense.
The closest consumer product I could find was from Wireless Loggers . This was a more expensive solution than I'd anticipated and the data was online but only accessible through a phone app. Sometimes things don't advance as fast as you'd expect. I was putting sensors inside cow's stomachs back in 2004 and watching temperatures remotely as they walked around eating and drinking using similar bespoke technology. Back then we were expecting this would be point and click easy and cost only a few dollars once sensor networks were commercialised, yet 12 years later it still looks difficult.

A pencil and paper was looking attractive but I'd also bought a convenient to use NodeMCU version of the esp8266 12-E microcontroller with integrated WiFi as I was reading about them everywhere as being functional and cheap, this one from Ali Express at just US$4.15 delivered.
The esp8266 12-E modules are tiny, cheap, lots of I/O and reasonably powerful.
Development is a lot more convenient when the esp8266 12-E is deployed in this NodeMCU format. It can be programmed directly from a USB port on a PC, powered from a USB charger during use and provides easy access to all I/O pins when plugged into a prototyping breadboard
Could I use this newfangled esp8266 to satisfy my curiosity?

After Christmas I got to spend a few days with Chris, a smart guy I've known since high school. He is living the dream, with the Hawkesbury River 3 metres from the door. It would be hard to get bored with that, but just in case he has a spare house at Chittaway Point, again with a wharf in the back yard. Between the crabbing, bush walking and watching boats pass from the balcony we had a go at learning how to use the esp8266.

The esp8266 first became available as an AT command driven serial modem but now several alternative development environments are available of which  LUA seems the most popular. We opted for Arduino which needs to be configured to program the esp8266 and the easiest way to get started is to follow the "Installing with Boards Manager" instructions. With Arduino, the standard method of testing if everything is set up correctly  is to compile and upload the code to blink a led on the target device. We found an esp8266 version of blink and in a few minutes were watching a blinking led. Trivial perhaps, but satisfying. A lot later there was simulated data periodically being uploaded to an online service.

We'd made more progress than I expected and I hadn't sourced a temperature sensor, so that had to wait until I returned home. A DS18B20 temperature sensor looks ideal at 5 units for A$3.88 and they are also available with a metre long waterproof cable for A$2.22. The DS18B20 reads temperature resolved to 9 bits by default and is configurable to 12 bits with an absolute accuracy of +/- 0.5 degrees. It needs only power, earth and a connection to a single GPIO pin. Multiple sensors can also be connected to the same pin. The DS18B20 documentation says a 4.7K ohm pull up resistor is required between the power and data pins and other implementations specify this also but the internal pull up in the ESP8266 12-E which is reported to range between 30K - 100K ohms has proved sufficient for me with two sensors connected physically close to the esp8266 12-E module. Later I came across a pedant aiming to Save The Earth One Resistor At A Time who analyses why a resistor is rarely required.
Connecting the DS18B20 temperature is this simple. Most documentation says a 4.7K pull up resistor is required between the data and power lines but the internal pull up resistor is good enough.

Using the sensor requires the Arduino temperature control library, which in turn requires the One Wire library for communication. It is possible to have installed a One Wire library previously so there might be two. If so the version that was installed with the other esp8266 libraries must be used.

We want to capture rapid temperature changes but minimise data streamed, so the algorithm I settled on is:-

  • read the temperature every 5 seconds, 
  • if it hasn't changed much go back to sleep, 
  • otherwise log measured temperature to a web service. 
If no data has been logged for a while usually the temperature hasn't changed but if I don't see new data I worry the logger has failed, so to provide reassurance, log at least every10 minutes.

After implementing this, when I have the sensor running, we can see live data:



or we can view it in a dashboard. Feel free to try this yourself, using our code. Before compiling, config-sample.h needs to be renamed to config.h and all the "TODO" fields replaced with appropriate values. For example, you must enter your WiFi access point name and password.

The problem was solved for only a few dollars if you put zero value on your time but this was as much a learning experience as a solution and I've seen enough for it to be already apparent why the esp8266 has generated so much interest. A couple of bucks for a tiny integrated microcontroller, with lots of I/O and inbuilt communications is what is required for the oft spoken of sensor networks to emerge on a scale proponents have envisaged. CSIRO spent many millions trying to develop something similar, marketed as a Fleck and now relegated to the sensor networks museum.  The esp8266 from Espressif Systems (Shanghai) is an order of magnitude more capable. That capability has attracted a buzz, will surely attract competitors and Espressif are close to releasing an even more capable esp32 that can additionally communicate over Bluetooth Low Energy.

I now know how hot the lounge has been though this is not yet a quality solution and there are others including a Lua version by Eyal Lebedinsky from our local hacker space. Ideally you want a device to arrive in the mail, fit a battery, look at a web browser and see the temperature. Getting a device that works out of the box and also comes with enough information to make tinkering easy would be nice. If time has value, the Wireless Loggers would have been much cheaper this time around but properly developed, the esp8266 could be at the core of a far superior solution and surely someone will do a commercial esp8266 based temperature logger soon.

Thursday, December 3, 2015

Texas Hold'em Is Scary and Fun

“I never put off till tomorrow what I can possibly do - the day after.” 
― Oscar Wilde

I played Texas Hold'em for a few years in the early 1990s at informal poker nights and had a lot of fun. Usually I went home a few hundred dollars up or down, mostly down to begin with.


Then I became a full time research student. Suddenly I had tremendous freedom in what I did each day with enough scholarship income to avoid hunger. It was a joyous time where I could pursue anything I found interesting and this gave me an opportunity to improve my game. I looked for information on strategy and odds. I found only meagre resources in the libraries I searched but consumed what I discovered. The internet was barely functional then, no web browser and no way to search but I did discover a no money online multiplayer Hold'em game and it was a great resource for procrastination. Gradually I improved so that I went home from poker nights in front more often than not. Not from a thorough understanding of probabilities and strategy but from an intuition developed through practice.

There is a lot of luck in no limit Holdem so winning or losing over a short period is not a good indication of skill but as I got better I formed the view I was second to one other player who, like me, turned up for every match. A poker night would last for hours but nearly all the money would be won in a few hands and the evening's outcome was determined in a few crucial seconds. I had worked with my nemesis when we were hired as contractors for a complex technical project. I considered he had insufficient humility to abandon a priori beliefs when evidence required it and an unwillingness to persevere when a problem became difficult which led too often to failing to solve the problems we were working on. We remained friends though, and he went on to a series of sales roles in high profile technology companies, excelled and became wealthy. I could often build up my pile during the grind but too often he took me on the big hands. I'd sit there quietly, drinking little and focusing on the game while he drank heavily, kept up a banter that included what was in his hand, his strategy and what we should all do. He was a great raconteur and sometimes played hands without bothering to look at his cards. He seemed easy to beat. He would tell you, who had the best hand then raise against them and as he predicted lose. As far as I could tell he never lied. There were several times he had nothing, told me so, raised against me and lost. I could trust him, would follow his advice and win. Yet too often on those few crucial hands, he took the pot. I don't think he bothered too much with theory or probability, he just got inside your head. The most painful experiences were when, confident I had the nuts, I'd  put all my effort into aggressively building the pot. I'd be pleased at my success only to discover my error at the showdown. The sense of being played by a drunk was emotionally devastating, I could never be sure if it was luck and that uncertainty further reduced my self belief. I no longer trusted my judgement and played too passively. Leaving money on the table is easier emotionally but over time is as expensive as overconfidence. My drunken nemesis would advise me to be more aggressive and while the advice was good it would unnerve me even more. Just occasionally he would be drunk enough to barely function and then I'd have a chance.

High volatility made outcomes for a particular evening difficult to predict but over time all players would form a view of their relative ranking and persuading those who ranked lowly to keep turning up is difficult. Some low ranking players, like me, would try harder and improve but most would give up. The best new players were those suffering the Dunning Krueger effect and volatility could hide this from them for quite a while. Even then though, a constant stream of new players was needed. Eventually we ran out and poker nights came to an end.

I find casinos entertaining. I don't delude myself I can win, but observing my own emotions and my success or otherwise in remaining objective and particularly watching the emotions in others is fascinating and fun.


With a 2.7% house edge and no skill required, roulette can provide a lot of fun as you interact with the dealer and other players, especially friends. If you turn over $1,000 you can expect to lose less than $30. So it is, on average, a cheap night out where you can go home with a wipe out or thousands up. Enough either way to generate an emotional response. Poker machines on the other hand I can't understand the attraction of. There is no fun in cold interactions with a machine that you know can't lose. The worst for me are those that provide choice. I know that if I hold reels optimally I will lose but if I get it wrong I'll lose more and yet the optimum strategy is unknowable. For one US machine optimum strategy has been documented but mostly you don't know how many symbols are on a reel, how many of the ones you want are there or even if the simulated reels are random. How do you deal with that? The impossibility of rational choice I find paralysing and distressing. I'm unusual though because:-
As much as 70 percent of the $48 billion in gaming revenues raked in by the casinos comes from slots. (Texas Hold'em poker and other table games may be the latest gambling fad ..., but for the casinos it's all about machines, machines, machines.) - The Atlanic 2005
With roulette, there is human interaction and thinking doesn't help, so you can ignore the game and have fun observing others and playing with your emotions and theirs. Elaborate pseudo strategies will impress friends when you win. You can wax lyrical on almost any topic and be taken seriously but you lose all respect when you wipe out. How weird is that when it's not your wisdom or lack thereof but down to variance? To maximise variance and therefore respect or contempt, bet on single numbers and walk away after a win.

In the Canberra Casino there is a Hold'em game. There are no women there and the players are mostly unkempt but these guys are the elite, the gods of the casino. I look over from the roulette table jealously, want to sit with them and recapture the fun of those poker nights of yore. Unlike roulette though, it is a game of skill. That table is no place for illusory superiority and the stakes are high enough to inflict serious injury. The internet resources available now are awesome though, so with sufficient effort it should be possible to develop the necessary skills. Odds calculators, play analysis software and realms of strategy advice. Of course the resources are available to everyone so the standard of play must be much higher than back in the day. No longer could Chris Moneymaker arise from obscurity and turn $40 into $2.5 million. Online practice is now a basic requirement rather than an opportunity to develop an edge. I'm not sure I could be bothered to put in the required effort but perhaps working at it with a friend would be fun. I believe I could have matched my nemesis in online play but face to face is a different, superior skill. I tried for a poker face but strong emotion is hard to conceal. Only face to face do you feel the depths of despair when you are owned or experience sheer joy if you successfully throw off the yoke. Only then are you happy when your opponent consumes another drink.

Could I do it?