AirVantage service failing to connect

Hey all,

I have a bit of a strange problem on my hands. One of our units is unable to connect to AirVantage despite having an active SIM and proper data connection. It fails with the following error and never recovers: Could not enable QMI M2M AVMS event reports. What makes this so strange is we have a fleet of 15 WP8548 modules, and so far only one unit is showing this problem. I can provide more logs if required but this seems to be the only “un-usual” log on this system.

Any help is greatly appreciated,
Cheers!

Meet the same problem with wp7607.

Try this
cm radio off
Wait 10s
cm radio on

This work for me. Network operator have changed and AV connection was good.

Ahoy there. We have had this same problem happen on a second unit today. Registered the unit in AirVantage a few hours ago, registered/activated the SIM card several hours ago, it has connected to 3 different cellular networks after “CM radio off” and “CM radio on”, but is not pushing data to AV.

We have about 22 units that have not experienced this problem. I’m going to try another SIM card as my next test.

Quick update, new SIM card did not resolve the issue.

Is this related to authenticating with AirVantage?

Here are some screenshots of logs.

One more update - The problem has appeared on three more units that we have tried to get online. So far we are 0 for 4 that are able to register on AirVantage and push data. All have the same error “Could Not Enable QMI M2M AVMS Event Reports”.

Hey @FrancisDuhaut,

Thanks for the suggestion. Unfortunately this did not work for us.

I could definitely be reading this wrong, but I’m focusing in on the “rc=-49” in the failure, and my quick take is that this corresponds to “Unknown Mandatory TLV”. Could you provide a little more detail on component versions? (Legato, linux baseline, modem FW). We can check if an incompatibility was introduced in this message that might cause a problem with a specific pairing of those components.

Thanks,
Ryan

Hey @rkirk,

We seem to have had some luck bringing these back to life by re-running the fdt2 based Windows firmware tool again. I’ve provided our firmware details below:

root@swi-mdm9x15:~# cm info
Device:                WP8548
IMEI:                  
IMEISV:                2E
FSN:                   
Firmware Version:      SWI9X15Y_07.12.14.00 r34472 CARMD-EV-FRMWR1 2017/11/29 18:24:42
Bootloader Version:    SWI9X15Y_07.12.14.00 r34472 CARMD-EV-FRMWR1 2017/11/29 18:24:42
MCU Version:           002.004
PRI Part Number (PN):  9907131
PRI Revision:          01.00
Carrier PRI Name:      GENERIC
Carrier PRI Revision:  001.034_000
SKU:                   1103453
Last Reset Cause:      Unknown
Resets Count:          Expected: 0	Unexpected: 0
root@swi-mdm9x15:~# uname -r
3.14.29ltsi-961ca71325_ed9f616cc8
root@swi-mdm9x15:~# legato version
18.04.0_02928cda04985dff10c72b699aef8ed2

@nvd, Do you think the issue started after a specific upgrade path was taken? For the devices that had connectivity restored by re-running the tool, did they remain stable in terms of connectivity? I’ve rebuilt 18.04.0 with the toolchain from Release 15, and I did immediately see a failure to connect, with the same error codes show in @coastalbrandon’s image:

Aug  1 22:20:48 swi-mdm9x15 user.err Legato: =ERR= | modemDaemon[678]/swiQmi T=m
ain | swiQmi.c swiQmi_CheckResponse() 813 | Sending QMI_WDS_START_NETWORK_INTERF
ACE_REQ_V01 failed: rc=0 (), resp.result=1.[0x01], resp.error=14.[0x0e]
Aug  1 22:20:48 swi-mdm9x15 user.err Legato: =ERR= | modemDaemon[678]/le_pa T=ma
in | pa_mdc_qmi.c StartSession() 1903 | Data connection failure Call End provide
d 45, Code 1
Aug  1 22:20:48 swi-mdm9x15 user.err Legato: =ERR= | modemDaemon[678]/le_pa T=ma
in | pa_mdc_qmi.c StartSession() 1914 | Data connection failure Verbose Call End
 provided Type 2, Verbose 204
Aug  1 22:20:48 swi-mdm9x15 user.err Legato: =ERR= | modemDaemon[678]/modemDaemo
n T=main | le_mdc.c le_mdc_StartSession() 1017 | Get Connection failure 45, 1, 2
, 204 

One limitation of pairing the newer AVC agent in Legato 18.04.0 with the older firmware from Release 15 is that you can’t have concurrent data connections with a USB host and a Legato application over the same APN. In my case, my Windows 10 host was automatically starting a data connection, which blocked my AirVantage test. As soon as I disabled the Windows connection, my AirVantage connection was successful. That was Release 15 Generic Modem + Linux, with LE 18.04.0 rebuilt with R15 toolchain. Is it possible that the QMI Event Reports failure isn’t the root cause? I’m not seeing that error though, which is strange, and I’d still like to get to the bottom of that error.

Thanks,
Ryan

Hey @rkirk,

On the units that were re-flashed they’ve stayed online for the most part. We do have 3 units that still will not connect tot he networks and I had put them to the side for now. In the logs I did see “Too many connections” for the modem, which I remember from when I would connect my computer to the MangOHs and they would act as a cellular modem for my computer’s internet connection. This would kill the AV connection, as you said. It’s definitely strange that I’m seeing that same “too many connections” error on the units that won’t connect to cell networks.

Brandon

Hey @rkirk,

I’m just testing with two units that were struggling to connect and stay connected to cellular networks.

Our units names are BRNKL 2004 and 2005, which both show the below log, which shows the “more than one application 2” lines, which I believe you were referring to previously.

Note, both of these units will go onto the cellular networks for several hours (push data, etc) and then they will mysteriously go offline then struggle to reconnect again. We have a test bench with about 15 other units that are working perfectly fine.

Aug  9 15:43:34 swi-mdm9x15 user.debug Legato:  DBUG | sensorToCloud[918]/sensorToCloud_exe T=main | _main.c main() 69 | == Starting Event Processing Loop ==
Aug  9 15:43:35 swi-mdm9x15 user.info Legato:  INFO | sensorToCloud[918]/sensors T=main | analoginputs.c analog_initSpiHandle() 77 | Initializing SPI handle
Aug  9 15:43:35 swi-mdm9x15 user.debug Legato:  DBUG | sensorToCloud[918]/framework T=main | le_spi_client.c DoConnectService() 363 | ======= Starting another client for 'sensorToCloud.sensors.le_spi' service ========
Aug  9 15:43:35 swi-mdm9x15 user.info Legato:  INFO | sensorToCloud[918]/sensors T=main | analoginputs.c analog_initSpiHandle() 81 | SPI open result: 0
Aug  9 15:43:35 swi-mdm9x15 user.info Legato:  INFO | sensorToCloud[918]/avPublisher T=main | avPublisher.c _avPublisher_COMPONENT_INIT() 1004 | sensorToCloud started
Aug  9 15:43:35 swi-mdm9x15 user.debug Legato:  DBUG | sensorToCloud[918]/framework T=main | LE_FILENAME le_mem_ForceAlloc() 841 | Memory pool 'framework.Default Timer Pool' overflowed. Expanded to 2 blocks.
Aug  9 15:43:35 swi-mdm9x15 user.debug Legato:  DBUG | sensorToCloud[918]/framework T=main | LE_FILENAME le_mem_ForceAlloc() 841 | Memory pool '.le_avdata_ClientData' overflowed. Expanded to 1 blocks.
Aug  9 15:43:35 swi-mdm9x15 user.warn Legato: -WRN- | dcsDaemon[826]/le_pa_dcs T=main | pa_dcs_linux.c pa_dcs_SetDefaultGateway() 568 | Default gateway or interface is empty
Aug  9 15:43:35 swi-mdm9x15 user.debug Legato:  DBUG | sensorToCloud[918]/framework T=main | LE_FILENAME le_timer_Start() 1260 | threadRecPtr->timerFD=16
Aug  9 15:43:35 swi-mdm9x15 user.debug Legato:  DBUG | sensorToCloud[918]/framework T=main | LE_FILENAME le_mem_ForceAlloc() 841 | Memory pool 'framework.FdMonitor' overflowed. Expanded to 11 blocks.
Aug  9 15:43:35 swi-mdm9x15 user.debug Legato:  DBUG | sensorToCloud[918]/framework T=main | LE_FILENAME le_mem_ForceAlloc() 841 | Memory pool 'framework.hashMap_refFdMonitors' overflowed. Expanded to 12 blocks.
Aug  9 15:43:35 swi-mdm9x15 user.warn Legato: -WRN- | modemDaemon[875]/le_pa T=main | pa_sim_qmi.c MapSimState() 858 | More than one application 2
Aug  9 15:43:35 swi-mdm9x15 user.warn Legato: -WRN- | modemDaemon[875]/le_pa T=main | pa_sim_qmi.c MapSimState() 858 | More than one application 2
Aug  9 15:43:35 swi-mdm9x15 user.info Legato:  INFO | modemDaemon[875]/le_pa T=main | pa_mrc_qmi.c pa_mrc_GetNetworkRegState() 2062 | called
Aug  9 15:43:35 swi-mdm9x15 user.warn Legato: -WRN- | modemDaemon[875]/le_pa T=main | pa_sim_qmi.c MapSimState() 858 | More than one application 2
Aug  9 15:43:56 swi-mdm9x15 user.debug kernel: [   60.891927] NMEA read driver miss interrupt, abandon current buff
Aug  9 15:44:12 swi-mdm9x15 user.debug kernel: [   76.891896] NMEA read driver miss interrupt, abandon current buff
Aug  9 15:44:20 swi-mdm9x15 user.info Legato:  INFO | supervisor[487]/supervisor T=main | supervisor.c HandleRebootExpiry() 524 | Expired reboot timer
Aug  9 15:44:24 swi-mdm9x15 user.warn Legato: -WRN- | modemDaemon[875]/le_pa T=main | pa_sim_qmi.c MapSimState() 858 | More than one application 2
Aug  9 15:44:26 swi-mdm9x15 user.info Legato:  INFO | gpsMonitor[863]/location T=main | location.c getLocation() 87 | Failed to get reading... retrying in 1 seconds
Aug  9 15:44:28 swi-mdm9x15 user.debug kernel: [   92.891683] NMEA read driver miss interrupt, abandon current buff
root@swi-mdm9x15:~#

Hi @coastalbrandon, I’ve been looking into this today and trying to find a connection to any known issues, on local networks or elsewhere. I haven’t found any additional background on this yet. I do think this is no longer an AirVantage connectivity issue, but a network rejection as described in the title of the other thread. If you agree, I think we can follow up on it there, since your initial logs there show a little more info about the nature of the problem. I’ll be responding in that thread shortly.

Hiya @rkirk & @coastalbrandon

We’ve recently been having issues with AV connections staying up over long periods. We’ve found that the link to AV will stop transmitting data even though the underlying network is still up (and we’re pushing data over the WWAN link using cURL).

I tried adding a ‘keepalive’ ping from the client device to AV to try and re-establish the link if it goes down. I started at 120 seconds, reduced oit to 30 seconds and in one case had to reduce it to 1 second intervals before we could reliably use AV to send commands to the client and keep the link up.

We are finding that no data was going over the LWM2M link between the client and AV, even though the AV monitor app in Legato had not been informed that the link had gone down, and even checking the AV connection state using AT commands showed that the modem thought the connection was still up.

We’ve submitted tickets to Sierra … but nothing back yet.

ciao, Dave