This issue happened when we tried to remove the application. So we came up with a test script:
count=0;while true;do count=$((count+1));echo "===Loop $count===";echo "---Updating MQTT app---";update MQTT_0009.ag35x.update && update -g && /usr/bin/app_reload;sleep 2;echo "------Updating MMTEST_SDK app-----";update MMTEST_SDK.ag35x.update && update -g && /usr/bin/app_reload;sleep 1;app status;sleep 2;app stop MMTEST_SDK;app stop TC_MQTT_0009;echo "------app installed ------";app status;sleep 2;echo "----app removed------";app remove MMTEST_SDK;sleep 1;app remove TC_MQTT_0009;sleep 1;echo "-----App status after Removal----";app status;done
After running certain iterations of this script, legato will crash. And the log printed from serial port:
Apr 25 06:44:14 mdm9607 user.crit Legato: *CRT* | MMTEST_SDK[3653]/framework T=main | LE_FILENAME TermSignalHandler() 643 | Terminated
[2023-04-26_10:39:02:826]Apr 25 06:44:14 mdm9607 user.crit Legato: *CRT* | TC_MQTT_0009[3606]/framework T=main | LE_FILENAME TermSignalHandler() 643 | Terminated
[2023-04-26_10:39:02:826]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | configTree[1118]/configTree T=main | nodeIterator.c OnTransactionTimeout() 142 | Write transaction <0x10001d1d> timer expired, for user root, <0>.
[2023-04-26_10:39:02:832]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | configTree[1118]/configTree T=main | treeUser.c tu_TerminateConfigClient() 517 | A fatal error occurred. Killing config session <0x812c409c> for user root, <0>. Reason: Transaction timeout.
[2023-04-26_10:39:02:835]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | updateDaemon[1119]/framework T=main | le_cfg_client.c SessionCloseHandler() 451 | Component for le_cfg disconnected
[2023-04-26_10:39:02:838]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | watchdog[1120]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (le_instStat:e160a983e92b4310ae8af3fbeff75a1b).
[2023-04-26_10:39:02:842]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | audioDaemon[1137]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (audioDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:02:842]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | supervisor[1103]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (le_instStat:e160a983e92b4310ae8af3fbeff75a1b).
[2023-04-26_10:39:02:845]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | atServerDaemon[1132]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (atServerDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:02:846]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | powerManagerService[1189]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (powerManagerService.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:02:849]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | portDaemon[1179]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (portDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:02:852]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | gpioService[1159]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (gpioService.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:02:854]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | fwupdateDaemon[1153]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (fwupdateDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:02:857]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | posDaemon[1184]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (posDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:02:860]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | secStore[1194]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (secStore.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:02:860]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | update[3680]/framework T=main | le_appRemove_client.c SessionCloseHandler() 451 | Component for le_appRemove disconnected
[2023-04-26_10:39:02:863]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | cellNetService[1142]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (cellNetService.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:02:865]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | dcsDaemon[1147]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (dcsDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:02:868]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | _UNKNOWN_[1452]/framework T=LegatoAudioThread | LE_FILENAME SessionCloseHandler() 451 | Component for le_audio disconnected
[2023-04-26_10:39:02:868]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | atClientDaemon[1131]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (atClientDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:03:009]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | modemDaemon[1173]/framework T=MrcProcessThread | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (modemDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:03:014]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | modemDaemon[1173]/framework T=SmsSendingThread | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (modemDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:03:018]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | modemDaemon[1173]/framework T=MdcEventThread | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (modemDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:03:021]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | modemDaemon[1173]/framework T=PulseRingSignalThread | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (modemDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:03:024]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | modemDaemon[1173]/framework T=ECallThread | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (modemDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:03:024]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | rSimDaemon[1174]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (rSimDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
[2023-04-26_10:39:03:027]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | _appStopClient[3681]/appStopClient_exe T=main | appStopClient.c _appStopClient_exe_COMPONENT_INIT() 53 | Error sending app name to the AppStop server socket. Connection refused
[2023-04-26_10:39:03:029]Apr 25 06:44:47 mdm9607 user.emerg Legato: *EMR* | _appStopClient[3684]/appStopClient_exe T=main | appStopClient.c _appStopClient_exe_COMPONENT_INIT() 53 | Error sending app name to the AppStop server socket. Connection refused
[2023-04-26_10:39:03:218]
[2023-04-26_10:39:03:218]INITzCC?[2023-04-26_10:39:03:233]INIT: Sending processes the TERM signal
[2023-04-26_10:39:03:424]init-eu-full stop sequence
We investigated into the issue, found that some file/directory went missing, next moment, legato got crashed. Later on, we added some print info for debug this issue, and then we observed the following error in the Snapshot.
Nov 1 00:07:17 mdm9607 user.crit Legato: *CRT* | MMTEST_SDK[2458]/framework T=main | LE_FILENAME TermSignalHandler() 643 | Terminated
Nov 1 00:07:18 mdm9607 user.crit Legato: *CRT* | TC_MQTT_0009[2412]/framework T=main | LE_FILENAME TermSignalHandler() 643 | Terminated
Nov 1 00:07:25 mdm9607 user.crit Legato: *CRT* | updateDaemon[1093]/framework T=main | LE_FILENAME file_Copy() 575 | eyelyn1 Attempting to copy a directory, '/legato/systems/current/config/modemService.scissors', into a file, '/legato/systems/unpack/config/modemServi
Nov 1 00:07:25 mdm9607 user.crit Legato: *CRT* | updateDaemon[1093]/framework T=main | LE_FILENAME file_CopyRecursive() 826 | eyelyn5 Attempting to copy a directory, '/legato/systems/current/config/modemService.scissors', into a file, '/legato/systems/unpack/config/m
Nov 1 00:07:25 mdm9607 user.err Legato: =ERR= | updateDaemon[1093]/updateDaemon T=main | system.c system_Snapshot() 791 | eyelyn file_CopyRecursive -1
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | configTree[1092]/configTree T=main | nodeIterator.c OnTransactionTimeout() 142 | Write transaction <0x10000c7b> timer expired, for user root, <0>.
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | configTree[1092]/configTree T=main | treeUser.c tu_TerminateConfigClient() 517 | A fatal error occurred. Killing config session <0x8155909c> for user root, <0>. Reason: Transaction timeout.
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | updateDaemon[1093]/framework T=main | le_cfg_client.c SessionCloseHandler() 451 | Component for le_cfg disconnected
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | watchdog[1094]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (le_instStat:e160a983e92b4310ae8af3fbeff75a1b).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | audioDaemon[1116]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (audioDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | supervisor[1077]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (le_instStat:e160a983e92b4310ae8af3fbeff75a1b).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | powerManagerService[1162]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (powerManagerService.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | gpioService[1136]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (gpioService.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | secStore[1167]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (secStore.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | fwupdateDaemon[1131]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (fwupdateDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | posDaemon[1157]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (posDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | update[2484]/framework T=main | le_appRemove_client.c SessionCloseHandler() 451 | Component for le_appRemove disconnected
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | cellNetService[1121]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (cellNetService.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | atClientDaemon[1110]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (atClientDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | atServerDaemon[1111]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (atServerDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | portDaemon[1152]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (portDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | dcsDaemon[1126]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (dcsDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | rSimDaemon[1147]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (rSimDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | modemDaemon[1146]/framework T=MrcProcessThread | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (modemDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | _UNKNOWN_[1434]/framework T=LegatoAudioThread | LE_FILENAME SessionCloseHandler() 451 | Component for le_audio disconnected
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | modemDaemon[1146]/framework T=SmsSendingThread | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (modemDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | modemDaemon[1146]/framework T=MdcEventThread | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (modemDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | modemDaemon[1146]/framework T=PulseRingSignalThread | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (modemDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | modemDaemon[1146]/framework T=ECallThread | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (modemDaemon.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | _appStopClient[2485]/appStopClient_exe T=main | appStopClient.c _appStopClient_exe_COMPONENT_INIT() 53 | Error sending app name to the AppStop server socket. Connection refused
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | _appStopClient[2486]/appStopClient_exe T=main | appStopClient.c _appStopClient_exe_COMPONENT_INIT() 53 | Error sending app name to the AppStop server socket. Connection refused
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | _appStopClient[2490]/appStopClient_exe T=main | appStopClient.c _appStopClient_exe_COMPONENT_INIT() 53 | Error sending app name to the AppStop server socket. Connection refused
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | _appStopClient[2492]/appStopClient_exe T=main | appStopClient.c _appStopClient_exe_COMPONENT_INIT() 53 | Error sending app name to the AppStop server socket. Connection refused
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | _appStopClient[2494]/appStopClient_exe T=main | appStopClient.c _appStopClient_exe_COMPONENT_INIT() 53 | Error sending app name to the AppStop server socket. Connection refused
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | _appStopClient[2496]/appStopClient_exe T=main | appStopClient.c _appStopClient_exe_COMPONENT_INIT() 53 | Error sending app name to the AppStop server socket. Connection refused
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | _appStopClient[2497]/appStopClient_exe T=main | appStopClient.c _appStopClient_exe_COMPONENT_INIT() 53 | Error sending app name to the AppStop server socket. Connection refused
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | _appStopClient[2498]/appStopClient_exe T=main | appStopClient.c _appStopClient_exe_COMPONENT_INIT() 53 | Error sending app name to the AppStop server socket. Connection refused
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | _appStopClient[2499]/appStopClient_exe T=main | appStopClient.c _appStopClient_exe_COMPONENT_INIT() 53 | Error sending app name to the AppStop server socket. Connection refused
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | smsInboxService[1172]/framework T=main | LE_FILENAME ClientSocketHangUp() 852 | Session closed by server (smsInboxService.smsInboxService.le_sim:f834c96427271edede285bc9174fc4f3).
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | _UNKNOWN_[1441]/framework T=TelcoStatusThread | LE_FILENAME SessionCloseHandler() 451 | Component for le_mdc disconnected
Nov 1 00:07:51 mdm9607 user.emerg Legato: *EMR* | _UNKNOWN_[1441]/framework T=LegatoMainThread | LE_FILENAME ClientSocketHangUp() 852 | Session closed
INIT: Sending processes the TERM signal1 mdm9607 user.emerg Legato: *EMR* | _appStopClie
init-eu-full stop sequence
killall: dlt-system: no process killed
killall: mm_mw_diag: no process killed
2282778 2023/04/04 22:28:02.196330 16262.9448 98 ATB4 SYS SYSL 0 log info verbose 1 <10>Nov 1 04:30:25 Legato: *CRT* | updateDaemon[1122]/framework T=main | LE_FILENAME file_CopyRecursive() 890 | Error reading file/directory information, '/legato/systems/current/config/modemService.paper'. (No such file or directory)
2282779 2023/04/04 22:28:02.196352 16262.9452 33 ATB4 DR STM 0 log info verbose 1 $GPRMC,,V,,,,,,,,,,N*53
2282780 2023/04/04 22:28:02.196357 16262.9463 99 ATB4 SYS SYSL 0 log info verbose 1 <11>Nov 1 04:30:25 Legato: =ERR= | updateDaemon[1122]/updateDaemon T=main | system.c system_Snapshot() 791 | eyelyn file_CopyRecursive -19
2282781 2023/04/04 22:28:02.196361 16262.9475 100 ATB4 SYS SYSL 0 log info verbose 1 <14>Nov 1 04:30:25 Legato: INFO | modemDaemon[1177]/le_pa T=main | pa_mrc.c sGetQmiRespSignalStrength() 510 | ===== gsm_sig_info_valid
2282782 2023/04/04 22:28:02.196366 16262.9487 101 ATB4 SYS SYSL 0 log info verbose 1 <14>Nov 1 04:30:25 Legato: INFO | modemDaemon[1177]/le_pa T=main | pa_mrc.c pa_mrc_GetSignalStrength() 613 | exit, rssi:-87
We suspect that “file_CopyRecursive” function has failed at this moment.
Could you please advice how can we resolve this issue?
Thanks a lot!
BR,
Louis