WP7607-1 - Legato doesn't start after reboot

Hi,

This issue happens very rarely, but if I reboot several times (using “reboot” cmd), I noticed that sometimes the legato framework won’t be running.

I’m running legato 18.06.3_dc55d84226223a3798b01c1a272e9603.

legato status:

root@swi-mdm9x28-wp:~/logread# legato status
Systems installed:
  4 [good] <-- current
Legato framework is NOT running.

logread | grep legato:

root@swi-mdm9x28-wp:~/logread# logread | grep legato
Jan  1 00:05:49 swi-mdm9x28-wp user.info kernel: [    6.703953] block ubiblock2_0: created from ubi2:0(legato)
Jan  6 00:05:49 swi-mdm9x28-wp user.notice root: update-ld-cache: Updating ld.so.cache using [/legato/systems/current/lib]...
Jan  6 00:05:52 swi-mdm9x28-wp user.warn Legato: -WRN- | _UNKNOWN_[785]/framework T=main | LE_FILENAME BindMount() 1450 | '/legato' is already mounted.

df -h:

root@swi-mdm9x28-wp:~/logread# df -h
Filesystem                Size      Used Available Use% Mounted on
devtmpfs                256.0K      4.0K    252.0K   2% /dev
/dev/mapper/rt           16.3M     16.3M         0 100% /
mdev                    256.0K      4.0K    252.0K   2% /dev
tmpfs                    76.3M         0     76.3M   0% /dev/shm
tmpfs                    76.3M         0     76.3M   0% /run
tmpfs                    76.3M     40.0K     76.2M   0% /var
tmpfs                    76.3M     24.0K     76.2M   0% /tmp
mdev                    256.0K      4.0K    252.0K   2% /dev
tmpfs                    76.3M         0     76.3M   0% /dev/shm
tmpfs                    76.3M         0     76.3M   0% /run
tmpfs                    76.3M     40.0K     76.2M   0% /var
tmpfs                    76.3M     24.0K     76.2M   0% /tmp
/dev/ubi3_0             104.5M      1.7M     98.1M   2% /mnt/flash
/dev/ubiblock2_0          3.8M      3.8M         0 100% /mnt/legato
/dev/ubi4_0               8.7M     68.0K      8.1M   1% /mnt/userrw
overlay                 104.5M      1.7M     98.1M   2% /etc
overlay                 104.5M      1.7M     98.1M   2% /data
overlay                 104.5M      1.7M     98.1M   2% /lib/firmware
/dev/ubi3_0             104.5M      1.7M     98.1M   2% /home
/dev/ubiblock1_0         24.8M     24.8M         0 100% /firmware
/dev/ubi3_0             104.5M      1.7M     98.1M   2% /legato
tmpfs                    76.3M     24.0K     76.2M   0% /etc/ld.so.conf
tmpfs                    76.3M     24.0K     76.2M   0% /etc/ld.so.cache
tmpfs                    76.3M     24.0K     76.2M   0% /etc/hosts
tmpfs                    76.3M     24.0K     76.2M   0% /etc/resolv.conf

I see here that two entries are missing compared to when the legato framework starts properly. The following two entries are at the end of the previous list when legato starts successfully:

cgroupsRoot              77.6M         0     77.6M   0% /sys/fs/cgroup
/dev/ubi3_0             104.5M      2.0M     97.8M   2% /legato/systems/current

Note that none of my apps are installed as I tried to reproduce with the base system.

If I do legato start, the framework will be launched and everything will work properly, so I’m considering having a script running outside of legato to make sure it’s always launched:

root@swi-mdm9x28-wp:~/logread# legato start
Starting Legato...
update-ld-cache: Updating ld.so.cache using [/legato/systems/current/lib]...
update-ld-cache: Updating /etc/ld.so.cache ...
update-ld-cache: Removing stale mount [/etc/ld.so.cache]...

Thanks for your help,
Best regards,
Ben

Hello,

Any news on this issue?

Thanks for your help,
Best regards

Hi @ben.o,
I think what you observe is linked to the rool back feature.
https://docs.legato.io/latest/conceptsConfig.html
when you update your device with a new system, this mechanism is used to recover the device in case an unexpected crash or resets or when the probation of the new system is reached.

in your case, if your device is running the “golden system”, and unexpected resets happen, this leads to stop the legato framework.

how many resets happen and with which frequency?
best regards,

Hi @plu

I’ve noticed this issue first using my system, then using the default system after erasing everything and putting back only official firmware and system.

Number of resets may vary, several between 10 and 50 I suppose. Frequency depends on me writing reboot every time I have a chance. I can usually reproduce in less than 30 min.

Thanks for your support,
Best regards,
Ben

Hi @ben.o,
regarding your use case I think you enter the case I described.
are you testing reset for stress tests purpose or is it the common way you proceed when dealing with the device?
best regards

Hi @plu,

Yes this test was for stress test purpose. It would be the common way though as I plan to use the reboot command from airvantage if needed.

Can this issue happen on a power on reset?

Thanks for your help,
Best regards,
Ben

Hi @ben.o,
yes this could happen on a power or reset , if several one occur within 60 seconds of boot-up;
Some improvements are included in more recent releases, I suggest you move on the last available one on the source website.
Best regards,

Hi @plu,

I tried to install the latest package when it went available (I’m using developer studio on windows). Unfortunately the package didn’t compile, I didn’t have time to dig into that so had to stay with the previous release.

Hi @ben.o,
do you remember which version it was?
Best regards

Hi @plu,

The version I failed to compile is 18.10.3.wp76xx-docker-201903041325.

The version I’m currently working with is 18.6.3.wp76xx-docker-201810311826 .

I’m using developer studio 5.3.1.201802071201 to download, install and compile.

Thanks for your help,
Best regards

Hi @ben.o,
we did not reproduce this issue. if you face again such behavior, could you please share with us the log from devstudio, to investigate?
best regards

compilationLogs.txt (83.2 KB)
Hi @plu,

I tried to create a new empty project using 18.10.3.wp76xx-docker-201903041325 and had the same issues than when I was trying to port my project, getting several Operation not permitted.

Please find the logs enclosed.

Thanks for your help,
Best regards

Hi @ben.o

this behavior was already reported to us.
it looks like the image is generated anyway in that case… please can you give a try and install it to verify if it’s functional anyway?

Hi @daav,

Is there any fix for the compilation problem using latest release?

The image generated seems functional but an error is generated at each compilation, I’m not sure about the impact of it.

Thanks for your help,
Best regards
Ben