Category Archives: Home Energy Monitoring

Munging my “Smart” Home Data

Posted on by

Context

For some of you who live in older homes, the feeling of too cold in the winter and too hot during the summer comes with the package. Last year, I thought of getting spray foam installed and figured now is the time to start analyzing the some of the data I’ve been collecting over the past three years. I told the installer that I had three years of data and would do a before and after test to see what the spray foam did in terms of performance. He stated he would purchase the data analysis. Needless to say one could just look at the heating bill to see if there is difference.  With variance in unit  costs of fuel, admin, etc., I did not want to bother normalizing that info. The geek in me wants to explore data mining and inference. So off I go to explore  Linear discriminant analysis and random forests.

For this exercise, I had three in-home temperature points, one outside, and several power related measurements. So far I had close to 3 million data points. I searched the web for an open source toolset that could help me with data analytics and decent plotting capabilities. Given I used the R programming language a few months back and liked the graphing capabilities in the ggplot2 package, it became my tool of choice. Note that Python is making in-roads in the data analysis space and for now, I want to remain focused on data analysis so R it is.

UDOO with SSD

Posted on by

I got around to set up the 120 GB SSD on the quad. I ended up making my own power cable with header wires to connect to +5 and Ground pins. I could not find a power connector that fit it J12 on the board at the local stores and didn’t want to order a small part online and pay for shipping.  What I have works.

The process was rather painless at first. I followed the procedure for creating a bootable image via OSX as described at the Udoo website. I then booted the board with the newly imaged SD card with the SSD connected to the board. I then installed gparted via

sudo apt-get install gparted

then ran

 sudo gparted

to partition the SSD and also create another partition on the SD card for backing up stuff since I had about 23 GB extra on the card to use.

I liked what another Udoo user suggested and allocated space for u-boot in the event that one day a complete boot from the SSD arrives. I created a partition for root/home and another for the swap as shown below.

 

Once the partition was complete, I did a quick check with fdisk -l to make sure the device was listed. I then mounted the SSD e.g. sudo mount /dev/sda1 /media/ssd for my environment. Then followed the instructions posted at elinux.org using the wget/tarball approach. The only thing left to prepare u-boot via setenv root… as the stated int the instructions. I noticed that my osx did not recognize the udoo quad plugged in the usb port and concluded I was missing a driver.  I ended up installing the osx version from Silicon Labs. Once that was complete, I proceeded with SerialTools to interrupt the boot process.

I set the root parameter as suggested and resumed the boot process. Under that scenario, I could not get the root system to point to the SSD. It was just booting like there was just an SD card. What I had to do was change the mmcroot to point as follows

mmcroot=/dev/sda1 root wait

rather than /dev/mmcblk0p1

I could not otherwise make the system root/home point to the SSD. I’m not a hard core linux person. This approach was not described as part of the instructions and I ended looking at the boot arguments to make this work.  The last touch was to set up /ect/fstab to deal with the swap partition and my backup space on the SD card.

The stab for my environment is the following:

UUID=332e43b0-97cd-42bb-b6a8-3429d4303bea none swap sw 0 0
UUID=9d07e200-9341-4278-8ad8-28b10212d8c5 /media/backups ext3 defaults 0 2

Note that the UUID is specific to my environment and can be found via the blkid command. I rebooted the board (sudo shutdown -r now) and then ran

df -h

and

swapon -s

and the output shown below.

ubuntu@udoo: -etc_003

 

I then installed mysql server as per standard Ubuntu instructions. What is left to do is to port what I have on my PC to the quad and replicate the SCADA side. Once that works, I will move the arduino code and energy monitoring hardware to the quad as well. Then call this project a day.

A Distraction – UDOO Quad

Posted on by

I received my UDOO Quad today. I was not expecting to dust off my faithful home energy system I wrote a couple of years ago. It has been running well yet I feel guilty of having a home computer running 24×7 to act as my SCADA host. The UDOO is the board that will blend both the Arduino and Linux in a nice board and allow Solid State Drive to hold all the data. On its own without a SSD it consumes around 3.7 watts in-standby.

I currently have about three years worth of energy and temperature data that I also don’t want to lose and it has to be migrated as well. I’m hoping to analyze it using the R-Language one day.  I’m hoping that it should be a relatively easy port as everything is cross platform.

UDOO Quad

Power Measurement Revisited

Posted on by

Spot Measurement

I finally got around to tinker with things again and opted to digress from the zigbee standalone mote-like development to revisit the power measurement.

I have three types of sensors to measure currents lying around and decided that given I already wrote the C++ classes and modbus integration for total home energy consumption, I could easily create a spot measurement modbus slave device to monitor specific loads.  Note that all my arduinos+zigbee are modbus slave devices and only the mote-like devices shall use a different protocol.

Continue reading →

Finished–Good Enough

Posted on by

Wiring to the Power Main

I finally got around to put in the current transformers  (CTs) in the panel last weekend. I had an electrician come and pull the meter out so I could install them. If you are wondering, I got permission from the utility company to do so. In hind-sight, I should have purchased split core transformers and just clip them on.

I can’t say much more about this, as it was quite a no brainer to do. This was the first time I tested  my Arduino with more than 13 amps and my only concern was if I was off in my scaling. Using commercial grade CT provide linear operation in the range I’m interested.

panel

I took a utilitarian view in packaging the Ardiuno and circuitry and ended up using a dead router enclosure to host the solution. With limited tools, I cut out a place for the LCD and xbee radio.  It works and is out of the way so it met my simple requirements.

mounted

Collecting the Data

I can collect power related information as shown below and log it over time for trending purposes. I can also collect reactive and apparent power but for power factor tells me the story at a glance. As stated in an earlier post, I used Mango to do all that.  I was posting stuff on Pachube and exceeded the posts allowed for free so I disabled it. I will revisit this later as this is not a high priority.

Also trended it the cumulative lbs of CO2 that is spewed out based on my consumption.  I reset the counter at midnight every night. I do the same with costs and kWatt-hr consumption.

If I had to Start Over

I would have wired and programmed the solution to handle updates over the xbee.  I settled for plugging the laptop in the device and load changes that way.

Next Project

I would like to stuff the server side software in a fanless PC like an ARTiGO. I must admit this stuff is boring and prefer to venture into more math/physic problems. Migrating the server side software is not really a project and more something I have to do  as one of my desktop PCs is doing the data collection.

The next project is to measure the temperature in different rooms and just use the XBee and a temperature transmitter. I also want to sense the on/off of key appliances as well over XBee. Alas, I don’t have the cycles to work on this as this is a part-time hobby.

So far I slapped together an Arduiono with XBee and transmit the temperature over modbus as illustrated in the diagram below.

Arduino as-built

The power monitoring Arduino has a slave ID of 2 and the temperature Arduino of 3. Using an Arduino to collect a temperature is overkill and is why I want to change it to be low power over a mesh network around the house. I may have a data concentrator over modbus to bridge the mesh with the host and something like the diagram below. Who knows.

Arduino to-be

xBee, Debugging, et al

Posted on by

Relatively Easy

I’ve been busy with work and never have the time to get back to this project. Anyway, I penciled in a few hours to wire up the xBee transmitter/receiver to the arduino. The xBee explorer is a time saver. What I wanted to do is to program the arduino over wireless as well. Given that time is limited, I opted to just send data over wireless to simplify connectivity.

The xBee explorer had 4 connection points that made it easy to hookup:

  • +5 to arduino +5
  • gnd to arduino ground
  • dout to rx (pin 1 of ardiuino)
  • din to tx (pint 2 of arduino)

That is pretty all that is required to wire up the xbee. The explorer takes care of the 5v to 3.3v level shifting.  Remember I am using the Duemilanove as the reference board. Note that you can’t plug in the USB in the Duemilanove and xBee in the explorer at the same time. The two will conflict with each other. This is rather trivial to hook up so no wiring diagram other than pin connection mapping is provided.

On the host side, I used the USB explorer which is basically leads to a plug and play model which I like.  Rather than regurgitate xBee configuration, I found that it is well explained here. (Xbee Configuration tutorial)  I tweaked the PAN, Baud rate, and type. I made the host a coordinator and the slave device a routing type. Again, explained well in the tutorial.

Just like the other parts of this project, I found soldering and wiring it up the hardest. I used left over wire wrap wires (I think the stuff is 20 years old) to connect things. I don’t wire wrap them and still solder. I like it because it is not tot bulky. It does break easy though.   Anyway, I have a renewed appreciation for those who package hardware. Although mine works, only a mother could love the way it looks. Alas, it is for my own use so prototype mode is good enough.

xbee

Current Transformer Revisited

I revisited the current transformer wiring in the context of safety. Rather than use terminal blocks, I should have used simple audio jacks to simplify the wiring. Either way, there was a safety issue that loomed in the back of my head and I wanted to address it before wiring the sensors to the power mains. The secondary in an open circuit mode of a transformer (not connected to anything) can reach a high voltage and should it come in contact to a person, guess who is going to be the resistor for that voltage to close the circuit. You.  It does not take much to kill you.

What I did was add a switch that shunts (shorts) the secondary of the transformer that I flip to “field work mode” before handling the terminals.  This is not like shorting a 120 AC line. The voltage on the secondary is the function of the reflection of the impedance on the primary. This can get mathematical so I will spare the math. Suffice it to say that shunting the secondary won’t hurt the current transformer and will add another level of safety.

Host Software

Posted on by

Rather than re-invent the wheel, I thought there should be software out there that provides SCADA/HMI functionality for free. It is a commoditized activity by now.  After combing the web, I stumbled and settled on Mango, an open-source M2M solution. Mango is Java based that integrates with MySQL and runs under Apache. All good stuff so far.  The web site describes the features and how to install the software.  I liked the data historian, alarms, and the various types data points including calling external web based sources. I set one to get the external temperature at the airport and using regex to scrape of the temperature form the HTML. All within Mango.

I found it relatively easy to get going. One thing I had to do was write a Modbus function 6 (write a single register) for the Arduino in C as I want to send commands to the arduino. i.e. set the time for example.

The diagram below shows some of the points I configured to handle the home energy monitoring. I used ISA motivated nomenclature to name the tags. I may revert to a human readable tags as I won’t have hundreds of points and becomes cryptic after a while.

mango3

Data logging for each data point is configurable as shown in the screen shot below. The example is for the temperature in the basement. All of this info is in the MySQL database from which one can chose to slice and dice the data later on using external tools.

mango5

As for graphical objects, one can create custom objects e.g. dials, meters, etc. and assign tags to them. When this is all done I will add an iPhone friendly UI to this so I can interact with the home energy system on the road. One thing that Mango does is allow me to work on my solution rather than re-inventing the wheel.  I know have the facility to hook up multiple ardduinos and focus on the fun stuff which is the embedded side of things.

Just XBee Left Wire

Posted on by

I finally got around to spend some time on this and got everything on a perfboard . I also included a temperature sensor (LM35) along with a button to select the LCD display mode.  If I had to change one thing, I would chose different connectors so I can disconnect the current and voltage sensing wiring without screwing on a terminal block. I removed the CT and voltage wiring to the terminal block in the picture.

IMG_2643

Given that I will be working with two current transformers any open circuit can be lethal. I will  add a switch that short circuits the secondary before I disconnect from the terminal blocks. The last piece to connect will be a XBee chip to send the data to the master. As stated early, I am using Modbus over serial as the protocol to communicate with the host.

One thing that did give me grief is that the temperature reading was erratic under certain scenarios.  I sampled the currents and voltage to get 2000 samples do some computations and then take a sample of the ambient temperature. If I did not perform the 2000 sample burst, I got a stable temperature reading. If I looped for sample 2000 readings then took a temperature, the temperature would be erratic.  I still don’t know why and my hunch is that it lies in the A/D mux in the Arduino.  What I did was take average 10 temperature readings with a delay of 20 ms between sample which led to a more stable temperature reading.

All that is left is to mount an XBee chip on the bread board and the hardware should be ready to stuff into an enclosure.

Host Communication

Posted on by

Modbus

Well I got around to dabbling with how to best communicate with a host to implement a data historian. I started writing a extremely dumbed down protocol inspired by Modbus  After a day I realized that why settle for less and figured that pure Modbus would be better.  I could then use Modbus master software to query my Arduinos.  I found a Modbus C code for the arduino to provide RTU slave functionality over serial line. I loaded the sketch and downloaded a Modbus tool to read some values from the arduino to see if it all worked. There is only support for Modbus function 3 and function 16 in the arduino implementation and I added code to handle function 6. If you know enough C it is rather straightforward.

Java Host

Now that connectivity using Modbus was feasible, I wanted to ensure that I could implement a data historian with open source tools and decided that Java was the way to go.  After all Apache Tomcat, mySQL, and the eclipse dev environment are free and just require wetware to use. I also wanted to have my own arduinoController to collect the energy information that abstracted the details of Modbus.  After combing the net, I quickly realized that there was no reference implementation from Sun or whatever they are called now for Win32 serial data com. I will use Linux later  host it all later, but wanted a platform independent solution.  I found RXTX as a potentially viable implementation and free! Now all I need is a Java implementation of a Modbus not surpisingly, that exists too at jamod.

What I hate is screen painting and UI work can be a  pain. I wanted a simple java application to test read/write of registers and display them in float, long, int, unsigned int, binary, and hex formats. It would be a way for me to understand the details further and I would not have to deal feel guilty using commercial software as a trial offer.  I digressed in my focus and looked for a tool that would allow me to create UI with minimal effort. I found JformDesigner to be flawless and will probably purchase it.  It installed in Eclipse without a glitch and a Luddite like me was productive in a matter of hours. Anyway after a couple of days, I had the RXTX source compiled and my arduinoController source integrated to provide a simple test tool for me to experiment with. The screen shot is shown below. The challenging part was how to display longs and floats as Modbus does not support floats and longs per say and is something one does at the application layer (for you OSI compliant people)

modbus tool

Representing Float

For those that are interested, to get implement a float from a Modbus register representation, one just has to do this

Arduino Side

It takes two registers ( 16 bytes in total to represent the float). One may ask how does one mash a float into ints on the arduino side? I used the union

union
{
int regsf[2];
float value;
} mashedFloat;

I used it as holding area to mash my floats into to registers.

e.g.  mashedFloat.value = powerfactor

regs[0] = mashedFloat.regsf[0];
regs[1] = mashedFloat.regsf[1];

Java Side

The data flows over standard Modbus to the host and on the Java side do something like this to unmash into a float.

float x = Float.intBitsToFloat( (mRegisters[i+1].toShort() & 0xffff) << 16 | (mRegisters[ i].toShort() & 0xffff) );

Modbus Registers

I captured how I will represent my power metrics from the main panel and send over Modbus. Rather than waste 2 registers for floats to represent something like 118.1 volts I will cast it into a single register as a 1181 and convert back on the host. The reason for this, on the Arduino Duemilanove, there is only 2k of RAM and I created my power calcs using classes and that is not cheap on memory.  I plan on taking the Modbus code I found on the web and make it class as well.  I will have other arduino’s in the house collecting spot power measurements and must ensure that I can upgrade code (read fix bugs)  in a consistent manner.

Tag Line Voltage Units Type R/W Offset
EI-011 Black Line Voltage Vrms short R 0
EIC-011 Black Line Voltage Crest Factor short R 1
CI-011 Black Line Current Irms short R 2
CIC-011 Black Line Current Crest Factor short R 3
CI-021 Red Line Current Irms short R 4
CIC-021 Red Line Current Crest Factor short R 5
KD-011 Black Delta T between Samples ms short R 6
KD-031 Red Delta T between Samples ms short R 7
TI-001 Local Temperature C short R 8
JI-011 Black Line Real Power Watt float R 9
JI-021 Red Line Real Power Watt float R 11
JIA-011 Black Line Aparrent Power VA float R 13
JIA-021 Red Line Aparrent Power VA float R 15
JIR-011 Black Line Reactive Power var float R 17
JIR-021 Red Line Reactive Power var float R 19
JIP-011 Black Line Reactive Power Factor float R 21
JIP-021 Red Line Power Power Factor float R 23
GC-001 Display Mode short W 30
GC-002 Number of samples per calculation short W 31
KC-001 Time sync (unix time) s long W 32

CO2 and kW*hr

On the arduino side, the CO2 produced and the kWatt*hr are accumulated over a 24 hour period and displayed on the LCD. I plan computing those on the host side rather than send over Modbus . I have all the raw data to do that and do statistical operations as well as part of the data historian. e.g. average CO2 per day, Standard Deviation, etc.

Prototyping – Part II

Posted on by

There is not much to this. A protoshield, the arduino, and a breadboard. Note the current transformer (donut). I have two of those to use in the panel.

IMG_1426

The first test was to plugging in a 60 W lamp to see what the measurement came too. I expected around 0.5 Amps and 60 watts. I was not disappointed.  I proceeded to plug in a toaster and put the ammeter in the circuit to see if my RMS current matched its RMS measurement. The photo below shows a .4% error. Not bad. Note the drop in the line voltage.

In Canada, the nominal line voltage is 120Vrms. I do measure the voltage as part of my power calculations and when I saw the 113 V I checked with the multi-meter and it read the same.  Assuming a 120 V reference would lead to errors in the power calculations. I should not be running a toaster outside my 20Amp line in the kitchen. I created a suicide cord that threads through the current transformer and is basically an extension cord. It plugs into a 15Amp line with other loads. 120 down to 113 is just over a 5% difference from the nominal line voltage. I am trying to rationalize why such a large dip. Anyway, the power measurement works.

IMG_1421

Crest Factor

The crest factor is the the ratio between peak and RMS signals. I do compute that and it gives me an idea on the shape of the waveform. A sinewave should have a crest factor of \sqrt{2} . The 60 W lightbulb had a crest factor of 1.40 for the voltage 1.40 for the current. Close enough.

I plugged in a variable speed drill and ran it a low RPM. As expected, the power factor went down to .27 with most of the power becoming reactive at 104 vars. The real power was just a mere 29.5 watts. The crest factor for the voltage was 1.39 and for the current, 3.96. That is expected as the duty cycle is changed to control the speed. For us home owners, we get charged for the real power consumed.  In industrial environments, the power company would penalize you for running with such an awful power factor.

I can’t wait to plug all this in the main panel see what the overall power consumption profile is. I expect the power factor to be closer to one.

Next Steps

Computing C02 emissions is trivial as well as projecting cost of power usage.  I would like to have that wired next to the power panel and displayed on the LCD sooner than later.  On the other hand, I need figure out the zigbee side of things as well as how to best do the data logging.  I can easily purchase another arduino later and focus on getting this prototype soldered on something more permanent.