Today I wanted to start experimenting with plotting data with Python, and began investigating the pandas library. I looked at a few tutorials, but quickly realized this wasn’t going to be a get-it-up-and-running-in-10-minutes thing, at least at my current skill level. I definitely have more reading to do on this topic.
Since I wasn’t ready to plot anything, and wasn’t sure what I wanted to work on next in rtlamrvis, I decided to work through a few Python tutorials. I’m going to start splitting my hour of code between tutorials and projects, until I have a better handle on the basics of Python. I think that will make my coding sessions more productive and less frustrating.
After working on tutorials for a while, I decided to look at rtlamrvis again, and changed the way it’s organizing the data from rtlamr. Instead of the list-of-dictionaries structure, I now have a dictionary-of-lists-of-dictionaries structure. Despite sounding more complicated, I think it will make it easier when it comes time to plot the data, since it will be grouped by ID rather than by date. Maybe there’s a way to do this all in a pandas dataframe, but I’m not at that level yet.
I used pprint, part of the standard library, to verify that the code was doing what I wanted. Here’s an example of the output, which shows the new data structure. (Note: I always change the IDs before I publicly post any of the output, since I’m running my script against real data received from utility meters in my neighborhood.)
{25130001: [{'Consumption': 1692966, 'Time': '2019-01-31T14:37:59.535801779-08:00'},
{'Consumption': 1692966, 'Time': '2019-01-31T14:39:01.912742779-08:00'},
{'Consumption': 1692966, 'Time': '2019-01-31T14:39:02.275644411-08:00'},
{'Consumption': 1692966, 'Time': '2019-01-31T14:40:02.637966157-08:00'},
{'Consumption': 1692966, 'Time': '2019-01-31T14:40:02.984066933-08:00'},
{'Consumption': 1692966, 'Time': '2019-01-31T14:44:08.92041054-08:00'},
{'Consumption': 1692966, 'Time': '2019-01-31T14:45:07.802489288-08:00'},
{'Consumption': 1692966, 'Time': '2019-01-31T14:45:08.149153195-08:00'}],
25130002: [{'Consumption': 1268989, 'Time': '2019-01-31T14:37:59.651609372-08:00'},
{'Consumption': 1268996, 'Time': '2019-01-31T14:39:02.027027329-08:00'},
{'Consumption': 1268996, 'Time': '2019-01-31T14:39:02.398141433-08:00'},
{'Consumption': 1269003, 'Time': '2019-01-31T14:40:02.753874744-08:00'},
{'Consumption': 1269003, 'Time': '2019-01-31T14:40:03.135628029-08:00'},
{'Consumption': 1269027, 'Time': '2019-01-31T14:44:08.69528619-08:00'},
{'Consumption': 1269027, 'Time': '2019-01-31T14:44:09.036069368-08:00'},
{'Consumption': 1269032, 'Time': '2019-01-31T14:45:07.918476317-08:00'},
{'Consumption': 1269032, 'Time': '2019-01-31T14:45:08.307969063-08:00'}],
25130003: [{'Consumption': 423976, 'Time': '2019-01-31T14:37:59.472225052-08:00'},
{'Consumption': 423976, 'Time': '2019-01-31T14:37:59.800811534-08:00'},
{'Consumption': 423970, 'Time': '2019-01-31T14:39:02.157949477-08:00'},
{'Consumption': 423970, 'Time': '2019-01-31T14:39:02.52558898-08:00'},
{'Consumption': 423963, 'Time': '2019-01-31T14:40:03.251006466-08:00'},
{'Consumption': 423939, 'Time': '2019-01-31T14:44:09.187101414-08:00'},
{'Consumption': 423934, 'Time': '2019-01-31T14:45:08.033166217-08:00'},
{'Consumption': 423934, 'Time': '2019-01-31T14:45:08.416413207-08:00'}],
25790004: [{'Consumption': 250917, 'Time': '2019-01-31T14:38:41.19477672-08:00'}],
32780005: [{'Consumption': 365170, 'Time': '2019-01-31T14:40:27.925069332-08:00'},
{'Consumption': 365170, 'Time': '2019-01-31T14:46:23.433415367-08:00'},
{'Consumption': 365170, 'Time': '2019-01-31T14:48:23.422122806-08:00'}],
33560006: [{'Consumption': 169636, 'Time': '2019-01-31T14:42:46.470004195-08:00'},
{'Consumption': 169636, 'Time': '2019-01-31T14:44:46.970037449-08:00'}],
34330007: [{'Consumption': 452616, 'Time': '2019-01-31T14:45:09.642549085-08:00'}],
34330008: [{'Consumption': 110476, 'Time': '2019-01-31T14:39:43.206066618-08:00'},
{'Consumption': 110476, 'Time': '2019-01-31T14:41:45.70258951-08:00'},
{'Consumption': 110476, 'Time': '2019-01-31T14:43:44.22001873-08:00'},
{'Consumption': 110476, 'Time': '2019-01-31T14:45:41.200623187-08:00'},
{'Consumption': 110478, 'Time': '2019-01-31T14:49:41.202099703-08:00'}],
35060009: [{'Consumption': 339538, 'Time': '2019-01-31T14:38:57.983626969-08:00'},
{'Consumption': 339538, 'Time': '2019-01-31T14:47:00.485356661-08:00'}],
41930010: [{'Consumption': 419058, 'Time': '2019-01-31T14:44:48.43359395-08:00'},
{'Consumption': 419058, 'Time': '2019-01-31T14:45:49.434384863-08:00'},
{'Consumption': 419058, 'Time': '2019-01-31T14:49:49.41852109-08:00'}],
47570011: [{'Consumption': 275208, 'Time': '2019-01-31T14:40:05.089249374-08:00'},
{'Consumption': 275208, 'Time': '2019-01-31T14:42:05.604592569-08:00'},
{'Consumption': 275208, 'Time': '2019-01-31T14:47:05.086830549-08:00'},
{'Consumption': 275208, 'Time': '2019-01-31T14:49:05.085712251-08:00'},
{'Consumption': 275208, 'Time': '2019-01-31T14:50:05.58677753-08:00'}],
47580012: [{'Consumption': 150236, 'Time': '2019-01-31T14:39:51.690447444-08:00'},
{'Consumption': 150236, 'Time': '2019-01-31T14:42:51.189222909-08:00'}],
49570013: [{'Consumption': 67188, 'Time': '2019-01-31T14:40:01.321136185-08:00'}]}
The data appears the way I expect, and it’s much easier to read now than in the original format from rtlamr. This is the output for 50 lines of input data. With more lines, you can start to see some interesting trends. For example, ID 25130003 shows a decreasing Consumption value, while the rest show an increasing value. When I look at a larger dataset, I can see that number fluctuate up and down. This would make no sense for a gas or water meter, so my guess is that 25130003 is a power meter and the owner has solar panels (sometimes they’re generating more power than they’re consuming). It’s a curiosity, but I’m not going to go around snooping at all my neighbors’ meters to solve the mystery. The point of this project is to help me identify my own meter, not invade others’ privacy.
