Watchdog timeout on atServerDaemon

I’m running into an issue where the module will crash seemingly random and if I look at the logging te cause seems to be the following:
Oct 21 08:09:41 swi-mdm9x28-wp user.crit Legato: CRT | watchdog[5319]/watchdogDaemon T=main | watchdog.c WatchdogHandleExpiry() 517 | proc 8735 [atServerDaemon] timed out

What would be a good direction to search in order to fix this issue?

Did you use atserver api in your application?

Yes, I’m using le_atServer_OpenBridge on /dev/ttyAT
le_atServer_AddDeviceToBridge to link it to /dev/ttyGS1

Saying that one i just realized I only notice this behaviour when plugged into my laptop via USB. That seems relevant.

Besideds that I create some commands and have them get parameters and intermediate responses and Finalresultcodes in the handling.

You can try this one and see if problem still exists.
If no problem is found, that means your code has problem.

I did see some differences in my code and theirs, using their set up for the TTY seems to have fixed it.

It still seems to block, return errors as it is busy and then crash if you mistype a command and use backspace to correct it. That is also in the example project, so I guess that’s just how it is.

How to reproduce with the test app?

I had to disable the check for setting the baudrate as that doesn’t appear to work on ttyGS1 by removing the goto error: on line 1127. I’ve also previously enabled the raw data uart over USB using AT!USBCOMP=1,1,0080049.

I build and installed it, then ran it using.

app runProc atServerIntegrationTest --exe=atServerTest – tty /dev/ttyGS1

then using teraterm send the following commands:

AT+BRIDGE="OPEN"
AT+BRIDGE="ADD"
ATI9
ATI<backspace><backspace>TI9
AT

and got the following responses:

AT+BRIDGE="OPEN"
+BRIDGE TYPE: PARA
+BRIDGE PARAM 0: OPEN

OK
AT+BRIDGE="ADD"
+BRIDGE TYPE: PARA
+BRIDGE PARAM 0: ADD

OK
ATI9
Manufacturer: Sierra Wireless, Incorporated
Model: WP7702
Revision: SWI9X06Y_02.35.02.00 5208b3 jenkins 2020/06/10 00:30:12
IMEI: 352653090119389
IMEI SV:  5
FSN: VU815670490110
+GCAP: +CGSM

OK
ATI9
Manufacturer: Sierra Wireless, Incorporated
Model: WP7702
Revision: SWI9X06Y_02.35.02.00 5208b3 jenkins 2020/06/10 00:30:12
IMEI: 352653090119389
IMEI SV:  5
FSN: VU815670490110
+GCAP: +CGSM
AT
ERROR

after this it gets stuck responding with ERROR to everything before crashing after ~2 minutes or so
See logging:

Oct 22 15:19:46 | supervisor[13221]/supervisor T=main | proc.c GetWatchdogAction() 362 | atServerTest@0 watchdogAction '' in proc section
Oct 22 15:19:46 | supervisor[13221]/supervisor T=main | app.c app_Start() 3471 | Starting app 'atServerIntegrationTest'
Oct 22 15:19:46 | supervisor[13221]/supervisor T=main | app.c CreateFileLink() 2084 | Skipping file link '/legato/systems/current/apps/atServerIntegrationTest/read-only/lib/libComponent_atServerTestComp.so' to '/legato/systems/current/ap
Oct 22 15:19:46 | supervisor[13221]/supervisor T=main | app.c CreateFileLink() 2084 | Skipping file link '/legato/systems/current/apps/atServerIntegrationTest/read-only/bin/atServerTest' to '/legato/systems/current/appsWriteable/atServer
Oct 22 15:19:46 | supervisor[13221]/supervisor T=main | proc.c proc_Start() 1403 | Starting process 'atServerTest@0' with pid 4412
Oct 22 15:19:46 | supervisor[13221]/supervisor T=main | supervisor.c SigChildHandler() 899 | Reaping unconfigured child process 4408.
Oct 22 15:19:46 | supervisor[4412]/supervisor T=main | proc.c proc_Start() 1363 | Execing 'atServerTest'
Oct 22 15:19:46 | supervisor[4412]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxCoreDumpFileBytes to value 8192.
Oct 22 15:19:46 | supervisor[4412]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileBytes to value 90112.
Oct 22 15:19:46 | supervisor[4412]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxLockedMemoryBytes to value 8192.
Oct 22 15:19:46 | supervisor[4412]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileDescriptors to value 256.
Oct 22 15:19:46 | supervisor[4412]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxMQueueBytes to value 512.
Oct 22 15:19:46 | supervisor[4412]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxThreads to value 20.
Oct 22 15:19:46 | supervisor[4412]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxQueuedSignals to value 100.
Oct 22 15:19:46 | atServerDaemon[4335]/framework T=main | safeRef.c le_ref_CreateRef() 547 | Safe reference map maximum exceeded for refFdMonitors, new size 42
Oct 22 15:19:46 | _UNKNOWN_[4412]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'atServerTest.atServerTestComp.le_atServer' truncated to 'atServerTest.atServerTestComp.l'.
Oct 22 15:19:46 | _UNKNOWN_[4412]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'atServerTest.atServerTestComp.le_mcc' truncated to 'atServerTest.atServerTestComp.l'.
Oct 22 15:19:46 | atServerTest@0[4412]/atServerTestComp T=main | atServerTest.c _atServerTestComp_COMPONENT_INIT() 1229 | ============== AT server test starts =================
Oct 22 15:19:46 | atServerDaemon[4335]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT+DEL', name length 44
Oct 22 15:19:46 | atServerDaemon[4335]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT+CLOSE', name length 44
Oct 22 15:19:46 | atServerDaemon[4335]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT+ABCD', name length 44
Oct 22 15:19:46 | atServerDaemon[4335]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT', name length 44
Oct 22 15:19:46 | atServerTest@0[4412]/framework T=main | safeRef.c le_ref_CreateRef() 547 | Safe reference map maximum exceeded for refle_atServer_ClientHandlers, new size 35
Oct 22 15:19:46 | atServerDaemon[4335]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'ATE', name length 44
Oct 22 15:19:46 | atServerDaemon[4335]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'ATA', name length 44
Oct 22 15:19:46 | atServerDaemon[4335]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'ATD', name length 44
Oct 22 15:19:46 | atServerDaemon[4335]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'ATH', name length 44
Oct 22 15:19:46 | atServerDaemon[4335]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT+BRIDGE', name length 44
Oct 22 15:19:46 | atServerDaemon[4335]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT+TEST', name length 44
Oct 22 15:19:46 | atServerTest@0[4412]/framework T=main | tty.c le_tty_SetBaudRate() 323 | Speed rate was not setted, 17/4098 not supported
Oct 22 15:19:46 | atServerTest@0[4412]/atServerTestComp T=main | atServerTest.c TtyLink() 1126 | Failed to configure TTY baud rate
Oct 22 15:19:46 | atServerDaemon[4335]/atServer T=main | le_atServer.c le_atServer_Open() 3309 | created device fd=11
Oct 22 15:20:01 | atServerTest@0[4412]/atServerTestComp T=main | atServerTest.c PrepareHandler() 339 | commandRef 0x20f1
Oct 22 15:20:01 | atServerTest@0[4412]/atServerTestComp T=main | atServerTest.c PrepareHandler() 346 | AT command name AT+BRIDGE
Oct 22 15:20:01 | atServerTest@0[4412]/atServerTestComp T=main | atServerTest.c PrepareHandler() 354 | Type PARA
Oct 22 15:20:01 | atServerDaemon[4335]/framework T=BridgeThread-0 | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'atServerDaemon.atServer.le_atClient' truncated to 'atServerDaemon.atServer.le_atCl'.
Oct 22 15:20:02 | atClientDaemon[4317]/framework T=atCommandClient-1 | thread.c PThreadStartRoutine() 416 | Failed to set thread name for atCommandClient-1 (34).
Oct 22 15:20:09 | atServerTest@0[4412]/atServerTestComp T=main | atServerTest.c PrepareHandler() 339 | commandRef 0x20f1
Oct 22 15:20:09 | atServerTest@0[4412]/atServerTestComp T=main | atServerTest.c PrepareHandler() 346 | AT command name AT+BRIDGE
Oct 22 15:20:09 | atServerTest@0[4412]/atServerTestComp T=main | atServerTest.c PrepareHandler() 354 | Type PARA
Oct 22 15:20:26 | atServerDaemon[4335]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'ATI', name length 44
Oct 22 15:20:33 | atServerDaemon[4335]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT^H^HTI9', name length 44
Oct 22 15:20:39 | atServerDaemon[4335]/atServer T=main | le_atServer.c ParseBuffer() 2564 | Command in progress
Oct 22 15:22:33 | watchdog[13236]/watchdogDaemon T=main | watchdog.c WatchdogHandleExpiry() 517 | proc 4317 [atClientDaemon] timed out
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | apps.c wdog_WatchdogTimedOut() 2429 | Handling watchdog expiry for: procId 4317
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | app.c app_WatchdogTimeoutHandler() 3943 | The watchdog for process 'atClientDaemon' in app 'atService' has timed out but there is no policy. The process will be restarted by default
Oct 22 15:22:33 | watchdog[13236]/watchdogDaemon T=main | watchdog.c CleanUpClosedClient() 355 | Client session closed
Oct 22 15:22:33 | atServerDaemon[4335]/framework T=BridgeThread-0 | le_atClient_commonclient.c ifgen_le_atClient_SetCommandAndSend() 1721 | Error receiving response from server
Oct 22 15:22:33 | watchdog[13236]/watchdogDaemon T=main | watchdog.c CleanUpClosedClient() 355 | Client session closed
Oct 22 15:22:33 | atAirVantage[4298]/framework T=main | messagingSession.c ClientSocketHangUp() 873 | Session closed by server (atAirVantage.atAirVantage.le_atServer:b7ddff0d499d71d3a23372d9447cbc78).
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcServer.c ClientCloseSessionHandler() 2182 | Client 0x7f5f73a0 closed, remove allocated resources
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcServer.c ClientCloseSessionHandler() 2182 | Client 0x7f5f6608 closed, remove allocated resources
Oct 22 15:22:33 | modemDaemon[13361]/modemDaemon T=main | le_mdc.c CloseSessionEventHandler() 961 | SessionRef 0x7f5fb740 has been closed
Oct 22 15:22:33 | dcsDaemon[13329]/dcsDaemon T=main | dcsServer.c CloseSessionEventHandler() 1329 | Client 0x7f5ab0e8 killed, remove allocated resources
Oct 22 15:22:33 | portDaemon[4316]/portDaemon T=main | le_port.c CloseSessionEventHandler() 2170 | SessionRef (0x7f573d80) has been closed
Oct 22 15:22:33 | portDaemon[4316]/framework T=main | messagingSession.c ClientSocketHangUp() 873 | Session closed by server (portDaemon.portDaemon.le_atServer:b7ddff0d499d71d3a23372d9447cbc78).
Oct 22 15:22:33 | watchdog[13236]/watchdogDaemon T=main | watchdog.c CleanUpClosedClient() 355 | Client session closed
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | proc.c proc_SigChildHandler() 2079 | Process 'atAirVantage' (PID: 4298) has exited with exit code 1.
Oct 22 15:22:33 | qmiAirVantage[4351]/framework T=main | messagingSession.c ClientSocketHangUp() 873 | Session closed by server (qmiAirVantage.qmiAirVantage.le_atServer:b7ddff0d499d71d3a23372d9447cbc78).
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcServer.c ClientCloseSessionHandler() 2182 | Client 0x7f5f7270 closed, remove allocated resources
Oct 22 15:22:33 | modemDaemon[13361]/modemDaemon T=main | le_mdc.c CloseSessionEventHandler() 961 | SessionRef 0x7f5fc910 has been closed
Oct 22 15:22:33 | dcsDaemon[13329]/dcsDaemon T=main | dcsServer.c CloseSessionEventHandler() 1329 | Client 0x7f5aafb8 killed, remove allocated resources
Oct 22 15:22:33 | atServerTest@0[4412]/framework T=main | messagingSession.c ClientSocketHangUp() 873 | Session closed by server (atServerTest.atServerTestComp.le_atServer:b7ddff0d499d71d3a23372d9447cbc78).
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | app.c app_SigChildHandler() 4072 | Process 'atAirVantage' in app 'atAirVantage' faulted: Restarting process.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | resourceLimits.c GetCfgResourceLimit() 207 | Configured resource limit maxStackBytes is not available.  Using the default value 0.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | proc.c proc_Start() 1403 | Starting process 'atAirVantage' with pid 5562
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | proc.c proc_SigChildHandler() 2079 | Process 'portDaemon' (PID: 4316) has exited with exit code 1.
Oct 22 15:22:33 | supervisor[5562]/supervisor T=main | proc.c proc_Start() 1363 | Execing 'atAirVantage'
Oct 22 15:22:33 | supervisor[5562]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxCoreDumpFileBytes to value 102400.
Oct 22 15:22:33 | supervisor[5562]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileBytes to value 102400.
Oct 22 15:22:33 | supervisor[5562]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxLockedMemoryBytes to value 8192.
Oct 22 15:22:33 | supervisor[5562]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileDescriptors to value 256.
Oct 22 15:22:33 | supervisor[5562]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxMQueueBytes to value 512.
Oct 22 15:22:33 | supervisor[5562]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxThreads to value 20.
Oct 22 15:22:33 | supervisor[5562]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxQueuedSignals to value 100.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | app.c app_SigChildHandler() 4072 | Process 'portDaemon' in app 'portService' faulted: Restarting process.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | resourceLimits.c GetCfgResourceLimit() 207 | Configured resource limit maxStackBytes is not available.  Using the default value 0.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | proc.c proc_Start() 1403 | Starting process 'portDaemon' with pid 5581
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | resourceLimits.c GetCfgResourceLimit() 207 | Configured resource limit maxStackBytes is not available.  Using the default value 0.
Oct 22 15:22:33 | supervisor[5581]/supervisor T=main | proc.c proc_Start() 1363 | Execing 'portDaemon'
Oct 22 15:22:33 | supervisor[5581]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxCoreDumpFileBytes to value 102400.
Oct 22 15:22:33 | supervisor[5581]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileBytes to value 102400.
Oct 22 15:22:33 | supervisor[5581]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxLockedMemoryBytes to value 8192.
Oct 22 15:22:33 | supervisor[5581]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileDescriptors to value 256.
Oct 22 15:22:33 | supervisor[5581]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxMQueueBytes to value 512.
Oct 22 15:22:33 | supervisor[5581]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxThreads to value 20.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | proc.c proc_Start() 1403 | Starting process 'atClientDaemon' with pid 5582
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | proc.c proc_SigChildHandler() 2079 | Process 'atServerDaemon' (PID: 4335) has exited with exit code 1.
Oct 22 15:22:33 | supervisor[5582]/supervisor T=main | proc.c proc_Start() 1363 | Execing 'atClientDaemon'
Oct 22 15:22:33 | supervisor[5581]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxQueuedSignals to value 100.
Oct 22 15:22:33 | supervisor[5582]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxCoreDumpFileBytes to value 102400.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | app.c app_SigChildHandler() 4072 | Process 'atServerDaemon' in app 'atService' faulted: Restarting process.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | resourceLimits.c GetCfgResourceLimit() 207 | Configured resource limit maxStackBytes is not available.  Using the default value 0.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | proc.c proc_Start() 1403 | Starting process 'atServerDaemon' with pid 5599
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | proc.c proc_SigChildHandler() 2079 | Process 'qmiAirVantage' (PID: 4351) has exited with exit code 1.
Oct 22 15:22:33 | supervisor[5582]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileBytes to value 5120000.
Oct 22 15:22:33 | supervisor[5599]/supervisor T=main | proc.c proc_Start() 1363 | Execing 'atServerDaemon'
Oct 22 15:22:33 | supervisor[5582]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxLockedMemoryBytes to value 8192.
Oct 22 15:22:33 | supervisor[5582]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileDescriptors to value 256.
Oct 22 15:22:33 | supervisor[5582]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxMQueueBytes to value 512.
Oct 22 15:22:33 | supervisor[5582]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxThreads to value 20.
Oct 22 15:22:33 | supervisor[5582]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxQueuedSignals to value 100.
Oct 22 15:22:33 | supervisor[5599]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxCoreDumpFileBytes to value 102400.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | app.c app_SigChildHandler() 4072 | Process 'qmiAirVantage' in app 'qmiAirVantage' faulted: Restarting process.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | resourceLimits.c GetCfgResourceLimit() 207 | Configured resource limit maxStackBytes is not available.  Using the default value 0.
Oct 22 15:22:33 | supervisor[5599]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileBytes to value 5120000.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | proc.c proc_Start() 1403 | Starting process 'qmiAirVantage' with pid 5614
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | proc.c proc_SigChildHandler() 2079 | Process 'atServerTest@0' (PID: 4412) has exited with exit code 1.
Oct 22 15:22:33 | supervisor[5614]/supervisor T=main | proc.c proc_Start() 1363 | Execing 'qmiAirVantage'
Oct 22 15:22:33 | supervisor[5614]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxCoreDumpFileBytes to value 102400.
Oct 22 15:22:33 | supervisor[5614]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileBytes to value 102400.
Oct 22 15:22:33 | supervisor[5614]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxLockedMemoryBytes to value 8192.
Oct 22 15:22:33 | supervisor[5614]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileDescriptors to value 256.
Oct 22 15:22:33 | supervisor[5614]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxMQueueBytes to value 512.
Oct 22 15:22:33 | supervisor[5614]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxThreads to value 20.
Oct 22 15:22:33 | supervisor[5614]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxQueuedSignals to value 100.
Oct 22 15:22:33 | supervisor[5599]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxLockedMemoryBytes to value 8192.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | app.c app_SigChildHandler() 4066 | Process 'atServerTest@0' in app 'atServerIntegrationTest' faulted: Ignored.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | app.c app_StopComplete() 4753 | app 'atServerIntegrationTest' has stopped.
Oct 22 15:22:33 | supervisor[13221]/supervisor T=main | apps.c DeactivateAppContainer() 374 | Application 'atServerIntegrationTest' has stopped.
Oct 22 15:22:33 | supervisor[5599]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileDescriptors to value 256.
Oct 22 15:22:33 | supervisor[5599]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxMQueueBytes to value 512.
Oct 22 15:22:33 | supervisor[5599]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxThreads to value 20.
Oct 22 15:22:33 | supervisor[5599]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxQueuedSignals to value 100.
Oct 22 15:22:33 | atClientDaemon[5582]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'atClientDaemon.watchdogChain.le_wdog' truncated to 'atClientDaemon.watchdogChain.le'.
Oct 22 15:22:33 | _UNKNOWN_[5599]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'atServerDaemon.atServer.le_atClient' truncated to 'atServerDaemon.atServer.le_atCl'.
Oct 22 15:22:33 | atServerDaemon[5599]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'atServerDaemon.watchdogChain.le_wdog' truncated to 'atServerDaemon.watchdogChain.le'.
Oct 22 15:22:33 | _UNKNOWN_[5562]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'atAirVantage.atAirVantage.le_atServer' truncated to 'atAirVantage.atAirVantage.le_at'.
Oct 22 15:22:33 | _UNKNOWN_[5562]/framework T=main | mem.c VerifyUniquenessOfName() 670 | Multiple memory pools share the same name '.le_avc_ClientThreadData'. This will become illegal in future releases.
Oct 22 15:22:33 | _UNKNOWN_[5562]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'atAirVantage.atAirVantage.le_avc' truncated to 'atAirVantage.atAirVantage.le_av'.
Oct 22 15:22:33 | _UNKNOWN_[5562]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'atAirVantage.atAirVantage.le_mdc' truncated to 'atAirVantage.atAirVantage.le_md'.
Oct 22 15:22:33 | _UNKNOWN_[5562]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'atAirVantage.atAirVantage.le_mrc' truncated to 'atAirVantage.atAirVantage.le_mr'.
Oct 22 15:22:33 | _UNKNOWN_[5562]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'atAirVantage.atAirVantage.le_data' truncated to 'atAirVantage.atAirVantage.le_da'.
Oct 22 15:22:33 | _UNKNOWN_[5581]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'portDaemon.portDaemon.le_atServer' truncated to 'portDaemon.portDaemon.le_atServ'.
Oct 22 15:22:33 | portDaemon[5581]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'portDaemon.watchdogChain.le_wdog' truncated to 'portDaemon.watchdogChain.le_wdo'.
Oct 22 15:22:33 | watchdog[13236]/watchdogDaemon T=main | watchdog.c GetConfigKickTimeoutInterval() 781 | No watchdog timeout configured for portService - using default 30000 ms
Oct 22 15:22:33 | portDaemon[5581]/portDaemon T=main | le_port.c JsonEventHandler() 1475 | JSON parsing is completed.
Oct 22 15:22:33 | _UNKNOWN_[5614]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'qmiAirVantage.qmiAirVantage.le_atServer' truncated to 'qmiAirVantage.qmiAirVantage.le_'.
Oct 22 15:22:33 | _UNKNOWN_[5614]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'qmiAirVantage.qmiAirVantage.le_avc' truncated to 'qmiAirVantage.qmiAirVantage.le_'.
Oct 22 15:22:33 | _UNKNOWN_[5614]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'qmiAirVantage.qmiAirVantage.le_mdc' truncated to 'qmiAirVantage.qmiAirVantage.le_'.
Oct 22 15:22:33 | _UNKNOWN_[5614]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'qmiAirVantage.qmiAirVantage.le_data' truncated to 'qmiAirVantage.qmiAirVantage.le_'.
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcFs.c ReadFs() 44 | failed to open /avc/fw/isInstallPending: LE_NOT_FOUND
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | packageDownloader.c packageDownloader_GetFwUpdateInstallPending() 609 | FW update install pending not found
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcFs.c ReadFs() 44 | failed to open /avc/sw/updateState: LE_NOT_FOUND
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcAppUpdate.c GetSwUpdateState() 1451 | SW update state not found. Returning 'INITIAL' state
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcFs.c ReadFs() 44 | failed to open /avc/sw/updateResult: LE_NOT_FOUND
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcAppUpdate.c GetSwUpdateResult() 1495 | SW update result not found. Returning 'INITIAL' result
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcFs.c ReadFs() 44 | failed to open /avc/sw/instanceId: LE_NOT_FOUND
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcAppUpdate.c GetSwUpdateInstanceId() 1342 | SW update instance id not found. Returning negative instance id.
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcFs.c ReadFs() 44 | failed to open /avc/sw/internalState: LE_NOT_FOUND
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcAppUpdate.c GetSwUpdateInternalState() 1386 | SW update internal state not found. Returning 'INVALID' state
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcAppUpdate.c avcApp_CheckNotificationToSend() 3356 | Invalid Instance ID
Oct 22 15:22:33 | _UNKNOWN_[5562]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'atAirVantage.atAirVantage.le_port' truncated to 'atAirVantage.atAirVantage.le_po'.
Oct 22 15:22:33 | _UNKNOWN_[5562]/framework T=main | mem.c VerifyUniquenessOfName() 670 | Multiple memory pools share the same name '.le_atServer_ClientThreadData'. This will become illegal in future releases.
Oct 22 15:22:33 | _UNKNOWN_[5562]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'atAirVantage.atThirdParty.le_atServer' truncated to 'atAirVantage.atThirdParty.le_at'.
Oct 22 15:22:33 | _UNKNOWN_[5562]/framework T=main | fdMonitor.c le_fdMonitor_Create() 449 | FD Monitor object name 'atAirVantage.atThirdParty.le_tpf' truncated to 'atAirVantage.atThirdParty.le_tp'.
Oct 22 15:22:33 | atAirVantage[5562]/framework T=main | safeRef.c le_ref_CreateRef() 547 | Safe reference map maximum exceeded for refFdMonitors, new size 42
Oct 22 15:22:33 | atServerDaemon[5599]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT+WDSC', name length 44
Oct 22 15:22:33 | atServerDaemon[5599]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT+WDSE', name length 44
Oct 22 15:22:33 | atServerDaemon[5599]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT+WDSG', name length 44
Oct 22 15:22:33 | atServerDaemon[5599]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT+WDSR', name length 44
Oct 22 15:22:33 | atAirVantage[5562]/framework T=main | safeRef.c le_ref_CreateRef() 547 | Safe reference map maximum exceeded for refle_atServer_ClientHandlers, new size 35
Oct 22 15:22:33 | atServerDaemon[5599]/framework T=main | safeRef.c le_ref_CreateRef() 547 | Safe reference map maximum exceeded for refle_atServer_ServerHandlers, new size 35
Oct 22 15:22:33 | atServerDaemon[5599]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT+WDSS', name length 44
Oct 22 15:22:33 | atServerDaemon[5599]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT+WDSI', name length 44
Oct 22 15:22:33 | atServerDaemon[5599]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT+WDSW', name length 44
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcFs.c ReadFs() 44 | failed to open /avc/fw/isInstallPending: LE_NOT_FOUND
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | packageDownloader.c packageDownloader_GetFwUpdateInstallPending() 609 | FW update install pending not found
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcFs.c ReadFs() 44 | failed to open /avc/sw/updateState: LE_NOT_FOUND
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcAppUpdate.c GetSwUpdateState() 1451 | SW update state not found. Returning 'INITIAL' state
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcFs.c ReadFs() 44 | failed to open /avc/sw/updateResult: LE_NOT_FOUND
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcAppUpdate.c GetSwUpdateResult() 1495 | SW update result not found. Returning 'INITIAL' result
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcFs.c ReadFs() 44 | failed to open /avc/sw/instanceId: LE_NOT_FOUND
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcAppUpdate.c GetSwUpdateInstanceId() 1342 | SW update instance id not found. Returning negative instance id.
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcFs.c ReadFs() 44 | failed to open /avc/sw/internalState: LE_NOT_FOUND
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcAppUpdate.c GetSwUpdateInternalState() 1386 | SW update internal state not found. Returning 'INVALID' state
Oct 22 15:22:33 | avcDaemon[13311]/avcDaemon T=main | avcAppUpdate.c avcApp_CheckNotificationToSend() 3356 | Invalid Instance ID
Oct 22 15:22:33 | QMUXD: WARNING qmi_qmux_if_pwr_up_init failed! rc=-6
Oct 22 15:22:33 | atServerDaemon[5599]/atServer T=main | le_atServer.c le_atServer_Create() 3387 | Create command 'AT+WDSTPF', name length 44

You can have a look here for further debugging

You can see if the backspace character is handled correctly.

It looks like there’s no boundary checking on the backspace character so it can underflow, which would explain the timeout:

                // backspace character
                else if ( input == 0x7F )
                {
                    devPtr->parseIndex--;
                }

I’ve now run into this issue when this error happened on a device that had not been connected to USB for at least days.

Is there a way to handle a crash in an application you have binded with? e.g. when atServer crashes I can disconnect the binding and then try to reconnect the binding without crashing my own app.

you can restart the process by faultAction defined in .adef file

There is no option to detect the sub-process crashing and redoing the binding without crashing my own process?

  • restart - log a critical message and restart the process.

Here says you can restart a particular process

This does not stop my own process from crashing when the subprocess crashs. I would like to keep my own process running at all times to reduce downtime and skip having to reconnect to our server.

Then why don’t you investigate why your app crashes?

Btw, does it work for other faultAction like stopping the app?
This can ensure you are really using the faultaction feature

the reason the app crashes is because the le_atServer runs into a timeout error which can occur due to e.g. someone pressing backspace too much.

The faultaction works as desired, if the application crashes it does indeed restart the application correctly. However I would like it so that atserver timing out would not crash the rest of the application.

Therefore my question is the following:
Is it possible to bind to a different IPC API in such away that if that sub proces crashes it will merely disconnect the IPC API and allow you to reconnect rather than crashing the entire application?

you can use IPC to communicate between two applications.
If one application crashes, it should not affect another one.
The application connecting to the web server should be the IPC server which does not need to bind to any other application.
The other application related to the le_atServer running into a timeout error should be the IPC client, and it will bind to the 1st application which is the IPC server.