How to record large amount of data when device is offline

#1

Hello all,

Using a MangOH Red board with WP7607 module, I tried to develop an app very similar to the RedSensorToCloud example from the MangOH github. The main difference is that I need to record a lot of sensor data even when my device is offline. For instance I need to record the accelerometer and gyro 3-axis every 100ms + GPS every 1sec during several hours.

I used the le_avdata_CreateRecord API to create my record. But after few seconds I can not add more data into it (le_avdata_Recordxxxx API returns LE_NO_MEMORY).
I tried to create a list of records, but again after few minutes or hours my application stops (getting “le_avdata_client.c SessionCloseHandler() 444 | ======= ‘recorder.avPublisherComponent.le_avdata’ service spontaneously disconnected ========” )

What is the best way to deal with this?

Based on the available RAM on the WP7607 I thought that I could keep more sensor data in RAM.

Or should I store the records in non-volatile memory? Is there a sample code that shows how to do so?

Thanks,
Julien.

#2

There are 2 solutions:
#1 accumulate your data outside of avdata record

#2 patch legato to be able to use more memory:

diff --git a/components/airVantage/avcDaemon/assetData.c b/components/airVantage/avcDaemon/assetData.c
index 423dfe42..d9b1c6ac 100644
--- a/components/airVantage/avcDaemon/assetData.c
+++ b/components/airVantage/avcDaemon/assetData.c
@@ -55,7 +55,7 @@
  * Maximum number of bytes for a string value field
  */
 //--------------------------------------------------------------------------------------------------
-#define STRING_VALUE_NUMBYTES 256
+#define STRING_VALUE_NUMBYTES 10000


 //--------------------------------------------------------------------------------------------------
@@ -63,7 +63,7 @@
  * Maximum number of bytes for CBOR encoded time series data
  */
 //--------------------------------------------------------------------------------------------------
-#define MAX_CBOR_BUFFER_NUMBYTES 1024
+#define MAX_CBOR_BUFFER_NUMBYTES 100000


 //--------------------------------------------------------------------------------------------------
diff --git a/interfaces/airVantage/le_avdata.api b/interfaces/airVantage/le_avdata.api
index a363bae..5481eaf 100644
--- a/interfaces/airVantage/le_avdata.api
+++ b/interfaces/airVantage/le_avdata.api
@@ -276,7 +276,7 @@ DEFINE PATH_NAME_BYTES = (PATH_NAME_LEN + 1);
  * Define the maximum characters and bytes of a string
  */
 //--------------------------------------------------------------------------------------------------
-DEFINE STRING_VALUE_LEN = 255;
+DEFINE STRING_VALUE_LEN = 100000;
 DEFINE STRING_VALUE_BYTES = (STRING_VALUE_LEN + 1);


diff --git a/interfaces/le_cfg.api b/interfaces/le_cfg.api
index bd9666f..847c63c 100644
--- a/interfaces/le_cfg.api
+++ b/interfaces/le_cfg.api
@@ -351,7 +351,7 @@ ENUM nodeType
  * Length of the strings used by this API.
  */
 //--------------------------------------------------------------------------------------------------
-DEFINE STR_LEN = 511;
+DEFINE STR_LEN = 28672;

 //--------------------------------------------------------------------------------------------------
 /**
--- a/3rdParty/Lwm2mCore/3rdParty/wakaama/core/block1.c
+++ b/3rdParty/Lwm2mCore/3rdParty/wakaama/core/block1.c
@@ -47,7 +47,7 @@
 #include <stdio.h>

 // the maximum payload transferred by block1 we accumulate per server
-#define MAX_BLOCK1_SIZE 4096
+#define MAX_BLOCK1_SIZE 100000

 uint8_t coap_block1_handler(lwm2m_block1_data_t ** pBlock1Data,
                             uint16_t mid,
--- a/apps/platformServices/airVantageConnector/avcDaemon/push.h
+++ b/apps/platformServices/airVantageConnector/avcDaemon/push.h
@@ -14,7 +14,7 @@
  * Maximum number of bytes for CBOR encoded data
  */
 //--------------------------------------------------------------------------------------------------
-#define MAX_CBOR_BUFFER_NUMBYTES 4096 // TODO: verify value
+#define MAX_CBOR_BUFFER_NUMBYTES 100000 // TODO: verify value

 //--------------------------------------------------------------------------------------------------
 /**
#3

Thanks Julien!

I tried the 2nd solution (patching Legato), but after few minutes each call to any le_avdata_Recordxxxx API takes longer and longer. After 2 minutes it takes more than 200ms.
Is that expected?

If it is, I will try to store my data outside of the avdata record.

Thanks,

#4

I stored my data outside of avdata record.
I initially tried to push the data using le_avdata_PushRecord API (by filling one or several small records just when I needed to push the data). But filling the records took too much time (~5sec for 60sec of recording).

So I am now trying to use the le_avdata_PushStream to push directly the binary data to AirVantage server. However I am not sure if I can then easily retrieve the binary data from the server.
From the timeline page, only some of the transmissions appears.
Using Postman I get this kind of output:

{
    "d421f50f99a04d6daecb3796cc7f036a": {
        "Recorder.sensors": [
            {
                "ts": 1539012235240,
                "v": "ޫETf\u0001"
            }
        ]
    }
}

Is it possible to retrieve the binary data using AirVantage API?

Thanks,

#5

@Julien_B

I have a similar situation and I’m curious if you managed to solve the issue?

Thanks,
Darren

#6

@dbeckwith

Sorry for the late reply.
I ended up storing my sensors data into files and I send those files periodically using le_avdata_PushStream API.
I also had to convert the sensors data into its ASCII representation in order to “see” it correctly on the AirVantage interface (and through its API)

Hope it helps.

Best Regards,
Julien.

#7

@Julien_B

Thanks for getting back to me and describing your solution. If you don’t mind me asking did you create your own file structure or did you use JSON or something like it?

#8

I created my own file structure.
Basically I stored the sensor data in raw format, without headers and so on, to optimize the file size as much as possible. On the server side, the decoding is based on the assumption that data won’t be corrupted and will match the expected scheme without any check apart from the file size itself.

#9

@Julien_B Thanks for the details!