PortDaemon Constantly Restarts in 19.04.0

Legato Application Framework
1

I am trying to migrate from Legato 18.09.0 to 19.04.0 on custom hardware and when I update the WP8548 with 19.04.0 the portDaemon constantly restarts every 30 secs, no errors or explanation are given except that the watchdog timed out.

Looking at the Legato AF repo on Github it appears the portDaemon changed quite a bit in April of 2019.

I have attached a small section of the log file.
Jul 2 21:15:29 swi-mdm9x15 user.info Legato: INFO | supervisor[710]/supervisor T=main | resourceLimits.c GetCfgResourceLimit() 207 | Configured resource limit maxStackBytes is not available. Using the default value 0. Jul 2 21:15:29 swi-mdm9x15 user.info Legato: INFO | supervisor[710]/supervisor T=main | proc.c proc_Start() 1401 | Starting process 'portDaemon' with pid 14913 Jul 2 21:15:29 swi-mdm9x15 user.info Legato: INFO | supervisor[14913]/supervisor T=main | proc.c proc_Start() 1361 | Execing 'portDaemon' Jul 2 21:15:29 swi-mdm9x15 user.info Legato: INFO | supervisor[14913]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxCoreDumpFileBytes to value 102400. Jul 2 21:15:29 swi-mdm9x15 user.info Legato: INFO | supervisor[14913]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileBytes to value 102400. Jul 2 21:15:29 swi-mdm9x15 user.info Legato: INFO | supervisor[14913]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxLockedMemoryBytes to value 8192. Jul 2 21:15:29 swi-mdm9x15 user.info Legato: INFO | supervisor[14913]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileDescriptors to value 256. Jul 2 21:15:29 swi-mdm9x15 user.info Legato: INFO | supervisor[14913]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxMQueueBytes to value 512. Jul 2 21:15:29 swi-mdm9x15 user.info Legato: INFO | supervisor[14913]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxThreads to value 20. Jul 2 21:15:29 swi-mdm9x15 user.info Legato: INFO | supervisor[14913]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxQueuedSignals to value 100. Jul 2 21:15:29 swi-mdm9x15 user.warn Legato: -WRN- | _UNKNOWN_[14913]/framework T=main | fdMonitor.c le_fdMonitor_Create() 734 | FD Monitor object name 'portDaemon.portDaemon.le_atServer' truncated to 'portDaemon.portDaemon.le_atServ'. Jul 2 21:15:29 swi-mdm9x15 user.warn Legato: -WRN- | portDaemon[14913]/framework T=main | fdMonitor.c le_fdMonitor_Create() 734 | FD Monitor object name 'portDaemon.watchdogChain.le_wdog' truncated to 'portDaemon.watchdogChain.le_wdo'. Jul 2 21:15:29 swi-mdm9x15 user.warn Legato: -WRN- | watchdog[726]/watchdogDaemon T=main | watchdog.c GetConfigKickTimeoutInterval() 781 | No watchdog timeout configured for portService - using default 30000 ms Jul 2 21:15:29 swi-mdm9x15 user.info Legato: INFO | portDaemon[14913]/portDaemon T=main | le_port.c JsonEventHandler() 1472 | JSON parsing is completed. Jul 2 21:15:59 swi-mdm9x15 user.crit Legato: *CRT* | watchdog[726]/watchdogDaemon T=main | watchdog.c WatchdogHandleExpiry() 517 | proc 14913 [portDaemon] timed out Jul 2 21:15:59 swi-mdm9x15 user.info Legato: INFO | supervisor[710]/supervisor T=main | apps.c wdog_WatchdogTimedOut() 2429 | Handling watchdog expiry for: procId 14913 Jul 2 21:15:59 swi-mdm9x15 user.crit Legato: *CRT* | supervisor[710]/supervisor T=main | app.c app_WatchdogTimeoutHandler() 3941 | The watchdog for process 'portDaemon' in app 'portService' has timed out but there is no policy. The process will be restarted by default. Jul 2 21:15:59 swi-mdm9x15 user.info Legato: INFO | watchdog[726]/watchdogDaemon T=main | watchdog.c CleanUpClosedClient() 355 | Client session closed Jul 2 21:15:59 swi-mdm9x15 user.info Legato: INFO | supervisor[710]/supervisor T=main | resourceLimits.c GetCfgResourceLimit() 207 | Configured resource limit maxStackBytes is not available. Using the default value 0. Jul 2 21:15:59 swi-mdm9x15 user.info Legato: INFO | supervisor[710]/supervisor T=main | proc.c proc_Start() 1401 | Starting process 'portDaemon' with pid 14971 Jul 2 21:15:59 swi-mdm9x15 user.info Legato: INFO | supervisor[14971]/supervisor T=main | proc.c proc_Start() 1361 | Execing 'portDaemon' Jul 2 21:15:59 swi-mdm9x15 user.info Legato: INFO | supervisor[14971]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxCoreDumpFileBytes to value 102400. Jul 2 21:15:59 swi-mdm9x15 user.info Legato: INFO | supervisor[14971]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileBytes to value 102400. Jul 2 21:15:59 swi-mdm9x15 user.info Legato: INFO | supervisor[14971]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxLockedMemoryBytes to value 8192. Jul 2 21:15:59 swi-mdm9x15 user.info Legato: INFO | supervisor[14971]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxFileDescriptors to value 256. Jul 2 21:15:59 swi-mdm9x15 user.info Legato: INFO | supervisor[14971]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxMQueueBytes to value 512. Jul 2 21:15:59 swi-mdm9x15 user.info Legato: INFO | supervisor[14971]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxThreads to value 20. Jul 2 21:15:59 swi-mdm9x15 user.info Legato: INFO | supervisor[14971]/supervisor T=main | resourceLimits.c SetRLimitValue() 301 | Setting resource limit maxQueuedSignals to value 100. Jul 2 21:15:59 swi-mdm9x15 user.warn Legato: -WRN- | _UNKNOWN_[14971]/framework T=main | fdMonitor.c le_fdMonitor_Create() 734 | FD Monitor object name 'portDaemon.portDaemon.le_atServer' truncated to 'portDaemon.portDaemon.le_atServ'. Jul 2 21:15:59 swi-mdm9x15 user.warn Legato: -WRN- | portDaemon[14971]/framework T=main | fdMonitor.c le_fdMonitor_Create() 734 | FD Monitor object name 'portDaemon.watchdogChain.le_wdog' truncated to 'portDaemon.watchdogChain.le_wdo'. Jul 2 21:15:59 swi-mdm9x15 user.warn Legato: -WRN- | watchdog[726]/watchdogDaemon T=main | watchdog.c GetConfigKickTimeoutInterval() 781 | No watchdog timeout configured for portService - using default 30000 ms Jul 2 21:15:59 swi-mdm9x15 user.info Legato: INFO | portDaemon[14971]/portDaemon T=main | le_port.c JsonEventHandler() 1472 | JSON parsing is completed.

Thoughts?
Darren

2

Hi @dbeckwith,

we did notice the same thing but haven’t identified the root cause just yet.
Seems like the accept(sockFd,) call in MonitorSocket hangs on 19.04.0

Weirdly enough we cannot reproduce the issue on current master branch, and on 19.05-release branch. So this issue will be fixed in 19.05.0, and we’ll backport the fix to 19.04-release branch once we identify the root cause.

3

@CoRfr Thanks for the update. I can also confirm that the issue isn’t in 19.02.0 which makes sense since the changes to le_port.c we introduced after 19.02.0 was released.

Do you have a timeline on when 19.05.0 will be released?

Thanks,
Darren

4

19.05.0 should be out the door within a week or so, as we are finalizing it this week.

5

19.05.0 was cancelled because the .05 didn’t make much sense anymore, we are now working on getting 19.07.0 before the end of the month.

6

@CoRfr Thanks for the update!