How to analyse household power to identify appliance usage

Grafana Live
1

Hi All, newbie here.

I’ve just installed InfluxDB v2.3.0 Ent. Edition (free) and Grafana v9.0.2. I’m using Telegraf to collect my household power in Watts every second.

I’ve started creating dashboards and all looks good.

Now, I’ve done this because I believe there are ways to analyse the data to determine when and how frequently a limited variety of appliances have been used and this is what I want to achieve.

Ideally, I’d output daily results to another time-series, which itself I could analyse for anomalies.

In reverse-order of complexity, these are the ones I’d like to identify:

Toaster
Kettle
Oven
Microwave
Washing machine

I’m new to data analysis and could use some pointers as to what to research, what tools to use and how to get started.

I’d also really like to find a mentor that I could discuss my project with and who could guide my journey, so any pointers to where to find one would be really appreciated.

I realise I may need to learn Grafana’s query language, Python and other tools but happy to do so.

Sorry if this is in the wrong topic. I couldn’t find one for absolute beginners…

Best regards and many thanks
Vitchling

2

Welcome

I think you already have what you need to do your analysis.

Can you please share what data points: fields in influxdb you are capturing ?

3

Hi Yosiasz, thanks for your reply.

I’m only capturing one data point in my house power dataset: “power”. It’s measured in watts and sent every second.

This shows my dashboard visualisation:
from(bucket: “openhabdb”)
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r[“_measurement”] == “mqtt_consumer”)
|> filter(fn: (r) => r[“_field”] == “power”)
|> filter(fn: (r) => r[“host”] == “OpenHAB”)
|> filter(fn: (r) => r[“topic”] == “OH2/ESP/HP/”)
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|> yield(name: “mean”)

A little reading around in one direction led me to a research paper comparing 10 different Non-Intrusive Load Monitoring methods, mostly using Machine Learning. A lot to dig into if I’m on the wrong track…

Cheers
Andrew

4

I personally love python. I would highly encourage you to go with a language you personally feel comfortable with and makes your entry into the big pool of data analysis easier. but also be cautious not to get lost in the rabbit holes. Maybe you might not even need to do python and analysis but find your answers just with telegrax, influxdb and grafana.

But to help guide you could you provide answers to the following.

What is the source telegraf is extracting data from? Can it provide you any data points that you use to identity appliances/watt consumer device? Kitchen iot!

5

Good advice about rabbit-holes!

Well, I can provide additional PIR data, including a kitchen PIR. I also feed these into a “presence” table in InfluxDB as the sum of each room’s detection over a 5 minute period. I had considered using these kitchen detections to cross-correlate appliance usage (though I don’t yet know how to do that). I expect it would work well for simple “on/off” devices like a toaster or kettle but only as a rough trigger-point for longer duration appliances such as a microwave or oven.

This shows my PIR dashboard:
from(bucket: “openhabdb”)
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r[“_measurement”] == “mqtt_consumer”)
|> filter(fn: (r) => r[“_field”] == “pircount”)
|> filter(fn: (r) => r[“host”] == “OpenHAB”)
|> filter(fn: (r) => r[“topic”] == “OH2/FF/BR/PIR/5min/” or r[“topic”] == “OH2/FF/MB/PIR/5min/” or r[“topic”] == “OH2/GF/Ki/PIR/5min/” or r[“topic”] == “OH2/GF/LR/PIR/5min/”)
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|> yield(name: “mean”)

If I can avoid having to learn ML or Python, that would be awesome :slight_smile:

6

Currently my Data Source for the House Power is an ESP32 microcontroller that has a clamp-coil on the main power feed into the house. I programmed it to use MQTT. I have a similar ESP32 device that I’m programming to use a PIR and MQTT with quite high re-trigger resolution but for now I’m using existing Zwave PIRs that are incorporated into my existing OpenHAB home automation system.

I’m hoping to come up with a commercial and probably cloud-based solution eventually (poss AWS with DynamoDB) or some other but ultimately, low compute resourceand scalability across users is good. Complexity is bad…

7

yeah keep it simple. so this is the approach I would recommend.
1 bucket that has it all. all your eggs in one basket :slight_smile:

time,value,field,measurement
2022-07-08T23:26:24Z,494,Toaster,das Küchenkaboodle
2022-07-08T23:26:24Z,494,Kettle,das Küchenkaboodle

or better

result,table,_start,_stop,_time,appliance,_value
my-result,0,2018-05-08T20:50:00Z,2018-05-08T20:51:00Z,2018-05-08T20:50:00Z,toaster,15.43
my
8

Hi Yosiasz

That looks like an ideal end-point - but I have no idea how I can get there from where I am…

…Unless I re-architect my device design to measure power individually at the plug with one per appliance, which would require lots of safety certifications, plus cost of a device per appliance.

I had hoped to avoid that with software that could recognise each appliance’s “signature” in the overall house power data.

Can you please explain what you had in mind?

9

not being familiar with ESP32 microcontroller/clamp-coil/OpenHAB, will need some help from you in that space.

Can you share an architectural design of your stack to help us identity the pipeline, if not maybe the following questions.
In the current microcontroller/OpenHAB/MQQT feed is there anything that gives you device “signature” ?

image
image1157×590 70.2 KB

based on

import pandas as pd
from influxdb_client import InfluxDBClient, Point, WritePrecision, WriteOptions
from influxdb_client.client.write_api import SYNCHRONOUS
from datetime import datetime
from dotenv import load_dotenv
import os
load_dotenv()

URL = os.getenv('URL')
TOKEN = os.getenv('TOKEN')
ORG = os.getenv('ORG')

bucket = "openhabdb"

"""
Write data into InfluxDB
"""
client = InfluxDBClient(url=URL, token=TOKEN, org=ORG, debug=True)
write_api = client.write_api(write_options=SYNCHRONOUS)  

col_list = ["_time","_value","_field","_measurement"]
df = pd.read_csv("./data/appliances.csv", usecols=col_list)

for index, row in df.iterrows():   
    _time =  row['_time']
    _field =  row['_field']
    _measurement =  row['_measurement']
    _value = int(row['_value'])

    point = Point(_measurement) \
            .field(_field, _value) \
            .time(_time, WritePrecision.NS)

    write_api.write(bucket, ORG, point) 
write_api.__del__()

"""
Close client
"""
client.__del__()

sample csv

_time,_value,_measurement,_field
2022-07-12 09:50:47.360,1.32,mqtt_consumer,Kettle
2022-07-12 10:50:47.360,2.64,mqtt_consumer,Cork Screw
2022-07-12 11:50:47.360,3.96,mqtt_consumer,Can opener
2022-07-12 12:50:47.360,5.28,mqtt_consumer,Coffe Maker
2022-07-12 13:50:47.360,6.60,mqtt_consumer,tooth brush - why do they call it tooth brush
10

“Currently my Data Source for the House Power is an ESP32 microcontroller that
has a clamp-coil on the main power feed into the house.”

This doesn’t help to answer your questions here at all, but would you mind
giving some more details of the hardware you’re talking about? I can guess
what sort of ESP32 device you’re using, but what have you managed to connect
it to in order to measure that level of power consumption?

I have some single-appliance devices of this sort from Sonoff, but I’d
certainly like to be able to put something on the entire house supply without
annoying my local supply company…

Thanks,

Antony.

1 Like
11

this looks amazing, didnt even know this whole world existed.

12

Hi Yosiasz

Sadly, no, there is nothing that identifies any appliance. I just have presence detection for the kitchen via PIR MQTT feed and aggregate power of the total house.

Ok, my system contains the following technology, all designed, programmed and under my control:

1 x House power monitor
4 x PIR detectors

I’ve designed the House power monitor as a small USB powered box containing a WiFi-enabled ESP32 microcontroller with some analogue to digital conversion. A small coil, shaped like a clamp fits around the main incoming power cable to a house and in short, it measures the total house power being used every second, then sends it over MQTT to an MQTT broker. In general, this device will connect to a user’s WiFi and in production the MQTT broker is likely to be Cloud-based, though for my development, I use my own.

Completely independently, I have designed a small USB powered PIR and temperature sensor, also using an ESP32 microcontroller. These also send PIR detections (and temperature) to an MQTT broker over WiFi.

Among other things, my project essentially tries to determine which of the appliances I mentioned earlier have been used through the day using their power “signature”. i.e., how much power they use when on and operating.

MQTT is a well established secure method of sending IoT data, with good AWS integration. In this development, Telegraph subscribes to the 4 PIR topics and 1 power topic and InfluxDB stores everything on one bucket.

To get to “I got the power” (I like that :slight_smile: ) I have to analyse the power time-series to recognise individual appliances. Eg: My kettle looks like this:

image
image1497×459 18.5 KB

Other devices have more complex power use patterns. My fridge and washing machine seem to have variable power and durations depending on what they are doing.
Fortunately, I don’t need to identify everything and can consider most appliances and always-on items as noise.

Does that help?

13

Hi Antony

I used this, ordered from eBay. I just looked again and they seem to have removed it…
image
You’d have to build it into your own circuit design though, and program it yourself. The only alternative I know of uses ZWave and a similar clamp, available from Aeotec through Amazon:
Aeotec Home Energy Meter Gen5, Z-Wave Plus Smart Electricity Usage Monitor, Report Real Time Power Consumption, 1 Clamp, Detect up to 60 amps for Single Phase System

14

Can these signatures be defined as ranges

15

Hi there. FYI There is a commercial product that claims to to what you are attempting. It’s called “Sense.” See sense.com. Might be of interest to you.

Cheers

16

Thanks for this. Very interesting. Clearly, they’re using AI. I wondered if I would have to go the same route. Sighs heavily in resignation as the unknown world of Machine Learning looms before me…

17

Hi Yosiasz

If we start simple, with just (say) a toaster, then I think we can say we know the power it uses and we know roughly how long it will be on for. Whether that can be defined as a range or not, I don’t know. I’m not sure what the term means in Influx. However, if we look at the dashboard of my kettle, it will look similar with a toaster, though the rise in power will be around 800w and the duration (say) 90 seconds.

Our analysis needs to recognise when the power value rises by approx. 800 and then falls by approx. 800 roughly 3 minutes later.

The same model should work for a kettle, but with different power and duration.

A more complex appliance, such as a microwave may exhibit a duty-cycle (repeating on/off) at a constant power but varying duration as it takes different times to defrost a meal. If cooking on full power, the duty-cycle may be 100%:0% (i.e. on all the time at full power).

I’m tempted to delve into Python for the first time as it may be easier to search for these patterns programmatically and the code snippets could be re-used when trying to analyse for (say) a microwave.

What do you think?

1 Like
18

Python is one route but I think it might be doable in grafana. Basically you are looking for signatures/patterns/fingerprint

If you define each appliance’s parameters/definitions like you did above the language used is inconsequential

19

The CEO claims that there are challenges that may go beyond voice recognition, since multiple devices (appliances) “talk*” at the same time.

*Talk → Use power at the same time.

If you have time here is a video interview link:

(RTT Episode 25: Sense Labs’ Mike Phillips on Bringing AI into the Smart Home - YouTube)

Cheers!

20

What you’re trying to do is called non-intrusive load monitoring. It’s about measuring the wire connecting the household electrical system to the grid. Non-intrusive means no separate measurement of appliances. The characteristics (timestamp, voltage, amps, powerfactor, harmonics, etc.) of the central wire are being measured and saved. Then the data is used to train a machine learning model. This model is used to measure loads/electric consumption of individual appliances. I can tell you that you need more than a simple clamp sensor to detect/discern individual appliances turned on simultaneously and predict loads/electric consumption of individual appliances. Take a look at the LSTM model for analyzing/predicting timeseries data.