11/13/2013

EFERGY E2 CLASSIC FSK TRANSMISSION DECODING USING A RTL-SDR DONGLE AND RASPBERRY PI

Finally!  A complete working prototype of an EFERGY E2 CLASSIC Transmission decoder using a Raspberry Pi and RTL-SDR USB Dongle


Decoding an Efergy Energy Meter with RTL-SDR and a Raspberry Pi


 
 
 
It is now possible to capture those 10s to 20s power consumption samples or Efergy E2 Transmission raw data through use of a RTL-SDR USB dongle connected to a Raspberry Pi. The Rspi runs on a fedora Linux OS with the C code decoding software installed.
 
Here are a couple of sample screenshots where the app reads data from the dongle using rtl_fm and a C program that decodes the baseband data and displays the computed values   in CSV format;
 












 











 




 Output data is printed in CSV format and can be captured and saved for excel use such as this one ;
 

 

The source code can be downloaded here,

A more detailed writeup of the software decoding can be viewed at Gough's Tech Zone

 

35 comments:

  1. Awesome work Nathaniel!

    I'm in Australia and own an Efergy Classic Elite 1.0R and I can confirm that your code works just a treat. I seem to find better reception with the E4000 (one packet every 6 seconds as set on the transmitter using a gain of 19.7 as suggested in your post) compared to the R820T (gain needs to be turned up to 30). Of course, changing it depending on your antenna situation etc makes sense.

    One note for people who are building the code - if you get problems with the function `pow' in the linker, use gcc -o EfergyRPI_001 EfergyRPI_001 -lm (the ordering of the switch -lm at the end seems to make the difference).

    I've compiled and run on Chrubuntu running on my Samsung ARM Chromebook.

    If you don't mind, I'll write up a post about it on my website linking to yours :)?

    ReplyDelete
    Replies
    1. Great! Sure please do post. The gain setting depends on a per dongle. Mine is a fitipower with only a maximum of 19.7 gain. The R820T can be set to a higher gain setting :)

      Delete
    2. I saw the post. Great writeup :) I kept the setup running for 24 hours and the only issue i got was the rtlsdr dongle stalling usually once or twice every hour. The reason behind was some conflict with the RT5370 wifi dongle - probably a raspberry 5v power issue (shared witht rtlsdr) or some software issue with the driver. This was fixed by detaching the wifi dongle during the capture. I may have to use the ethernet port or probably a powered usb hub ?

      Delete
    3. Ah, that might be right! USB packet losses will cause rtl_fm to starve or hang. I'm running my E4000 on the Chromebook and I've surpassed 24h with zero stalls and zero losses of packets (one was received each and every 6 seconds). Fantastic work!

      I did modify your code with my (very nasty) C skills to do direct logging to file name given as an argument, and to change line endings to DOS format, and to moderate the flush to file times to avoid wearing out the eMMC on my Chromebook. I hope you don't mind, full credit to you of course :).

      I also find that some of the TV tuner dongle crystals or PCB might be to blame. I've had a few that drop out on all computers after some time (normally a few hours) and need to be unplugged and replugged to get going again. I'm convinced it's a hardware issue as it disappears using another dongle of the same make.

      I suppose it's always going to need a few tweaks. I'm still trying to see whether there's any particular setting which works better for the R820T and makes it receive more reliably in the long run. I get stalling on my R820T within 3 minutes all the time, and I'm still not sure why it would be the case as the USB is handled by the RTL2832U and has nothing to do with the front end technically. Might just be an unlucky dongle that doesn't work well with my Chromebook. I might try another dongle later.

      I'll keep you posted on any further discoveries, but needless to say, I'm plenty happy with what I've got already. I'd be very interested to see how you actually achieved this and what resources you used in reverse engineering and coding a decoder for this.

      Delete
  2. An update - I just posted some of my findings to help people get themselves on the way to "better" decoding. I've found that the tuned frequency is critical due to tuner crystal offsets - a value between 433.5 and 433.6Mhz should work best, trying in 10khz steps. Further to that, using a solid core piece of wire for the antenna rather than using the stock antenna, and turning *down* the gain might also improve SNR. But best to visualize *exactly* what your algorithm is trying to decode before running it (my mistake!).

    More information, see http://goughlui.com/?p=5122

    Also, as a good sign, the code runs happy on my Dell Inspiron 640m Laptop running Lubuntu 13.10. :)

    ReplyDelete
  3. You know what? I've even surprised myself. I've managed to get it working in Windows!

    http://goughlui.com/?p=5129

    Sorry for the spam-level of comment frequency.

    ReplyDelete
    Replies
    1. Wow! You did it! I was initially trying it on windows using the precompiled rtl_fm.exe and somehow the passing of data from rtl_fm to the exe decoder somewhat stalls. Not sure what happened there thus the transfer to linux :)

      Delete
    2. I did notice there's a slight bug in the precompiled rtl_fm in the display of frequency and sample rate, but it seems to be only cosmetic. I tried to build my own, and failed miserably because I tried building it in Cygwin with the provided libusb which is hopeless.

      I did, however, discover another reason for intermittent tuning - one of my MMCX connectors must have worn from all the plugging and unplugging, so when the dongle heats up, it loses contact with the centre pin. A slight nudge of the antenna connector brings back decoding without unplugging/closing the program. Just *another* possibility amongst many.

      Still can't say everything's bug free, but I might be switching over to Windows for primary logging as my print/storage server which is on 24/7 runs Windows :).

      Delete
  4. Posted an Excel result of a Fridge running ... http://rtlsdr-dongle.blogspot.com.au/2013/11/efergy-e2-classic-10-second-sampling.html :)

    ReplyDelete
  5. Looks like you might have a point in regards to the Windows decoding performance - I did a comparison (amongst other things in a random posting):
    http://goughlui.com/?p=5142

    No long term "unrecoverable" stalls, and I've been over on the Windows/Cygwin build for 24h+ now.

    ReplyDelete
  6. That was one great progress going on there :) I think in my case on the windows platform, it has something to do with the passing of data (through windows stdout-pipe-stdin) from rtl_fm to the decoder. Not sure what is happening there ...

    ReplyDelete
  7. Hi again!

    Just hoping to get your input and or any assistance - I recently just got an efergy HM01 which doesn't decode using the e2/elite compatible code (http://goughlui.com/?p=5161). I've analyzed the transmission in that post finding it has a different FSK format and a different frequency.

    I then made some controlled load visualizations of the FSK signal (http://goughlui.com/?p=5184) where I've done my best to identify what the portions of the signal are, but without further analysis, I'm still not entirely sure how the data is encoded.

    I was wondering whether you could illuminate the process of decoding and correlating the data, or whether there's anything else I need to be aware of before delving further into it.

    As an aside, I made a collateral finding during that experiment which seems to suggest a reason why Windows decoding lags behind Linux, and it seems it may be related to rtl_fm Windows Build itself having USB buffer losses/problems. The finding is preliminary and I've not tried hard to make it happen consistently.

    ReplyDelete
  8. we need more data, we just need to determine which one is the id and which one is the ADC value. Post two raw captures of a zero load, and couple (about 5) of with load raw captures plus the actual power data value being displayed on the screen .

    ReplyDelete
  9. No problem. I just did a capture of as many power levels as I could as indicated by the LCD. Unfortunately, due to my set-up, I'm rather limited in how high I can go in the range, I know it's not ideal. The manual says the CT is specified for up to 90A in each phase (21.6kW indicated), but the highest I could do in this capture was 1.472kW. Each raw file contains just one packet, except for zero which contains three packets (all identical). You can download the 16.4Mb zip file at http://goughlui.com/static/hm01-rawsamples.zip

    Thanks in advance for all your help!

    ReplyDelete
  10. I just remembered it was a "current transformer" and by looping the wires around a few times, I've got a second batch of samples which cover up to 15.27kW of indicated load on the LCD http://goughlui.com/static/hm01-rawsamples2.zip (size: 9.43Mb). This should make it easier to spot exactly how big the ADC value is.

    If there's anything in particular you need me to do, give me a shout :).

    Many thanks,
    Gough.

    ReplyDelete
  11. My receiver states that it is an Elite 2.0R, and I am using all three clamps to measure all three phases entering my house. When buying the thing I was hoping to be able to monitor each phase separately.

    I am using rtl-sdr-compatible sticks with R820T tuners, which works just fine for me, given that I find the right ppm value and by trial and horror find a working gain value. My rtl_fm command for the stick I am currently using is:

    rtl_fm -f 433.55e6 -s 192e3 -r 96e3 -g 28.0 -d 1 -p 25

    For some strange reason (considering my setup with three clamps for three phases) your program, EfergyRPI_001, actually does give the same numbers as those shown on the display, even though it is just using part of the decoded data. I would have thought that you code would just give the power used on a single phase.

    I compared your decoding to the one suggested by Joe Desbonnet at http://jdesbonnet.blogspot.se/2010/09/smart-electricity-meter-based-on-efergy.html, and if his interpretation of the data is correct I should have one byte with the high nibble being 0x4 (battery OK, 6 second interval). I do, but not in the same place as he does, for me it is byte 4 of 8 by your count, corresponding to byte 8 of 12 by Joe's count (including 3 sync bytes and constant 0x00 byte).

    Does anyone know how to find the average current per phase?

    Johan Adler
    Stockholm, Sweden

    ReplyDelete
  12. I just disconnected one sensor at the time from the transmitter, saw no obvious change in the output. If this version only transmits the total average current for all phases I will have to make a system of my own using those clamps to measure each phase.

    ReplyDelete
  13. If the values are the same even on 3-phase then i assume that the computation for the combined 3-phase 3 adc data is being done on the transmitter side. As such, there is no separate bit definition for each phase input on the transmitted binary data.

    ReplyDelete
  14. Hi, I'm trying to decode that device connecting it to an arduino directly on the AMICCOM A7201A data_out pin... I correctly retrieve an 8 byte sequence every 6 seconds, but i cannot decode that. i've tried the formula on your code (and many other) but it doesn't work :-( ...
    Here is a sampling of received bytes (in decimal format) with the corresponding value on the efergy monitor:

    232:71:220:64:165:202:10:44 --> 479W
    232:71:220:64:51:229:10:21 --> 519W
    232:71:220:64:50:215:10:6 --> 512W
    232:71:220:64:40:43:10:80 -->499W

    Any Idea about the decoding?
    Note:
    first 4 bytes never change (only 64 changes into 128 after link missing)
    the 10 i believe is a multiplier, @2200W it becomes 100

    ReplyDelete
    Replies
    1. maybe still an issue on the fsk decoding, 7th byte (bytes[6] in my code) values should be something like -2,-1,0,1,2,3. Where you able to verify using the checksum code ?

      Delete
    2. Francesco. Would you mind to share your arduino code pls ?

      Delete
    3. Sorry for my incredible delay... Now the arduino code is
      done (now i have to test if it keep working on long time), there is a difference of some watt between lcd and arduino but I believe is a rounding problem between the two electronics... E2 data_out is connected to arduino digital pin 3 by a logic-level-shifter (3.3 to 5v), this makes readings more easy, E2 gnd to arduino gnd and E2 power to arduino 3.3v (E2 shows low battery but at 5v i've some problem)... And there is the code:


      int in = 3;
      int i=0;
      int incomingTime[67];
      char bytearray[9];
      int startComm=0;
      int processingTime;
      char bytedata;
      int bitpos;
      int bytecount;
      int dbit;
      double current_adc;
      double watt;
      int limit = 67;
      boolean flag;

      //Voltaggio
      int VOLTAGE=220;

      void setup() {

      Serial.begin(115200);
      pinMode(in, INPUT);
      Serial.println("Initial Delay");
      delay(1000);
      Serial.println("DONE");
      }

      void loop() {

      processingTime=pulseIn(in,HIGH);
      if (processingTime > 400){
      i=0;
      startComm=1;
      }

      if (startComm==1){
      incomingTime[i]=processingTime;
      //Serial.println(i);
      i++;
      }
      if (i == limit){

      startComm=0;
      i=0;
      flag=1;

      }


      if (flag == 1){
      //Serial.println("flag");
      for (int k=1;k<=limit;k++){
      if (incomingTime[k] != 0) {
      //Serial.println(incomingTime[k]);
      if (incomingTime[k] > (20) ){
      dbit++;
      bitpos++;
      bytedata = bytedata << 1;
      if (incomingTime[k]>100){
      bytedata = bytedata | 0x1;
      }
      if (bitpos > 7){
      bytearray[bytecount] = bytedata;
      bytedata = 0;
      bitpos = 0;
      bytecount++;
      }
      }
      }
      }
      if (bytecount == 8){
      //Serial.println("Byte");
      flag=0;
      if ((calculate_watts(bytearray))==1){
      Serial.println("Checksum OK - ");
      Serial.print("ADC Sensore: ");
      Serial.println(current_adc);
      Serial.print(" -- Watt: ");
      Serial.println(watt);
      Serial.print(" -- Ampere: ");
      Serial.println(watt/VOLTAGE);
      memset (incomingTime, -1, sizeof(incomingTime));
      memset (bytearray, 0, sizeof(bytearray));
      delay(100);
      }
      else{
      flag=0;
      Serial.println("Checksum KO");
      memset (incomingTime, -1, sizeof(incomingTime));
      memset (bytearray, 0, sizeof(bytearray));
      delay(100);
      }

      }
      //Serial.println("-----------------------------------");

      //Other Code


      }
      else{
      bitpos = 0;
      bytecount = 0;
      dbit = 0;
      bytedata = 0;
      }
      }


      int calculate_watts(char bytes[])
      {

      char tbyte;

      int i;

      tbyte = 0;

      for(i=0;i<7;i++) {
      //Serial.println(bytes[i],BIN);
      tbyte += bytes[i];
      }
      tbyte &= 0xff;
      if (tbyte == bytes[7])
      {
      current_adc = (bytes[4] * 256) + bytes[5];
      watt = (VOLTAGE * current_adc) / ((double) 32768 / (double) pow(2,bytes[6]));

      return 1;
      }

      ;

      return 0;
      }

      Delete
    4. For stability, i've found that it works better with:
      processingTime=pulseIn(in,HIGH,10000);
      instead of:
      processingTime=pulseIn(in,HIGH);

      Delete
  15. Hi all!

    I'm decoding my Chacon ecowatt metter using your code but sometimes a get wrong data like:


    01/15/14,09:37:03,2338.256836

    01/15/14,09:37:09,13026521696844833065935256648602352487567081900689719673491072120580162453504000.000000

    01/15/14,09:37:15,144.996643

    In this case is only one read but sometime i get many times in a row.

    Perhaps I have to try another rtl_fm configuration ?

    ReplyDelete
    Replies
    1. There is a bug in the code bytes[6] should be treated as signed number. Thus change the line affected by the computation to ...

      double fact;

      if (bytes[6] > 0x80)
      {
      fact = ((0xff - bytes[6]) + 1) * -1; /* Make a negative from a signed byte */

      }
      else

      fact = (double) bytes[6];

      result = (VOLTAGE * current_adc) / ((double) 32768 / (double) pow(2,fact));

      Delete
    2. bytes[6] values should be something like 3,2,1,0,-1,-2,-3 . You will get that error if the monitor receives a low wattage data where bytes[6] variable should be a negative number :)

      Delete
    3. The issue isn't bytes[6] - char is already signed, so the code additions you gave there do nothing at all. Compile with clang and you'll find clang's static code analysis even points out that fact.

      The issue is that bytes[4] and bytes[5] are also signed, when they should be treated as unsigned. If bytes[4] is over 0x80 it would result in negative numbers, while if bytes[5] is over 0x80 it wouldn't result in a massive difference, but it would be subtly different (lower by a few watts for example) than what the receiver reads.

      The good news is it's easily fixed in one line with a bit of type casting. Simply replace the line that sets current_adc with the following:

      current_adc = (((unsigned char*)bytes)[4] * 256) + ((unsigned char*)bytes)[5];

      There are probably more elegant ways of doing that, but it works.

      Delete
  16. I can't download the source code. It keeps insisting I install ilivid executable. I finally caved and did that and I STILL can't get it. What's the secret?

    ReplyDelete
  17. User Gough Lui has posted a modifed code from his blog site. Maybe if you can try downloading it from his site ?

    ReplyDelete
  18. The link to the source is broken :(

    ReplyDelete
  19. Does anyone have the original piece of source code for raspberry pi and efergy e2?
    the link is not working and i acnnot find it elswhere:(

    ReplyDelete
  20. Does anyone know if this code is suppose to work with e2-ir?
    I get the following data when running: rtl_fm -f 433.55e6 -s 192e3 -r 96e3 -g 28.0 -d 1 -p 25|./EfergyRPI_log -a 1



    Found 1 device(s):
    0: Realtek, RTL2838UHIDIR, SN: 00000001

    Using device 0: Generic RTL2832U OEM
    Found Rafael Micro R820T tuner
    Tuner gain set to 28.00 dB.
    Tuner error set to 25 ppm.
    Tuned to 433838000 Hz.
    Oversampling input by: 6x.
    Oversampling output by: 1x.
    Buffer size: 7.11ms
    Sampling at 1152000 S/s.
    Output at 192000 Hz.

    Efergy Power Monitor Decoder - Running in analysis mode using verbosity level 1


    Analysis of rtl_fm sample data for frame received on 10/14/15,20:11:06
    Number of Samples: 1367
    Avg. Sample Values: -10631 (negative) 3604 (positive)
    Wave Center: -3513 (this frame) 0 (last frame)
    Decode from positive pulses: 09 2e 14 40 00 00 00 00 chk: 8b kW: 0.000

    ReplyDelete
  21. For Efergy e2 Classic and Efergy IR/Optical have a look at rtl_433 here https://github.com/merbanan/rtl_433

    Using device 2: Generic RTL2832U OEM
    Found Rafael Micro R820T tuner
    Exact sample rate is: 250000.000414 Hz
    [R82XX] PLL not locked!
    Sample rate set to 250000.
    Bit detection level set to 8000.
    Tuner gain set to Auto.
    Reading samples in async mode...
    Tuned to 433550000 Hz.
    2016-08-02 14:41:42 : Efergy Optical
    Power KWh: 0.112 KWh

    ReplyDelete
  22. I'm thinking of setting up an Efergy e2 to monitor the current on the three phases. Is it possible to use Tellstick to receive the data? How to present ampere isntead of watt (it is actually VA not watts as there is some reactive power involved). Are there any web GUI available to present the data e.g. a skript that generates HTML-pages with day/week/month data?

    ReplyDelete
  23. Interesting project! I came here searching for info about the Efergy TPM and now I'm wondering if this is doing essentially the same thing as the Efergy Engage Hub (decoding and analyzing the data off the monitor) http://efergy.com/us/products/energy-gateways/elite-true-power-meter-engage-hub#.WNAQH28rKUk

    ReplyDelete