Legato not installed using fwupdate

Legato Application Framework
1

Hello,

We have encountered an issue using fwupdate on a unit deployed to a customer. This was only seen once after thousands of successful uses, but we need to understand what can be done to either recover or workaround it.

This is on a WP7610 module, based on Sierra Release 16.1 (Legato 19.11.6).

The spk that we use during the update contains the linux distribution, modem firmware, and legato framework including our applications.

fwupdate downloadOnly is used first, then several hours later fwupdate install is used.

The issue observed was that the linux distritbution was installed successfully, but not the legato framework and apps. This led our product to stop working properly without a way to recover.

Looking at the logs, no error were seen during the install that would indicate the issue. However, /mnt/legato is empty.

Is there a way to troubleshoot this further to understand why the install failed?

Now that fwupdate isn’t available anymore, is there a way to reinstall the legato framework in order to recover?

Thanks for your help,
Best regards

2

do you see any fwupdate binary?
BTW, what is status of legato framework?


root@fx30:~# find / -name "fwupdate"
/data/le_fs/fwupdate
/mnt/flash/ufs/data/le_fs/fwupdate
/mnt/legato/apps/f4d8baa0a5ca46871166afa697dd8e2e/read-only/bin/fwupdate
/mnt/legato/system/bin/fwupdate
find: /proc/3348: No such file or directory
root@fx30:~# legato status
Systems installed:
  23 [good] <-- current
Legato framework is running.
3

Hello, I find the following:

root@hhv2-009280:/mnt/flash/legato# fwupdate
-sh: fwupdate: not found
root@hhv2-009280:/mnt/flash/legato# legato status
-sh: legato: not found
root@hhv2-009280:/mnt/flash/legato# find / -name "fwupdate"
/data/le_fs/fwupdate
/legato/systems/current/appsWriteable/tools/bin/fwupdate
/mnt/flash/ufs/data/le_fs/fwupdate
/mnt/flash/legato/systems/current/appsWriteable/tools/bin/fwupdate
find: /proc/30659: No such file or directory
find: /proc/30677: No such file or directory
4

if you cannot use fwupdate, then probably the only thing to recover is to use:

  1. an external PC to run fdt2.exe or swiflash to download your image
  2. use AVMS for FOTA upgrade

BTW, have you compared the UART2 boot up message with OK to see what happens

5

So this means that using fwupdate has a chance of bricking any unit? This is very concerning for a product deployed in the field.

Is there a way to get more understanding on why it happened to have a chance to prevent it?

BTW, have you compared the UART2 boot up message with OK to see what happens

Sorry, are you referring to the boot serial output, such as the one in dmesg?

6

yes, UART2 debug port should show more detail
You can compare with a good module

7

Ok - yes unfortunately those are the logs I was looking at, but there was no error suggesting a failure in installing legato.

Is there any known causes for this? I can see this issue has happened before: /mnt/legato Empty

Has there been an internal ticket opened for it?

8

Since the rate of reproducibity is so low (<0.1%) , it is better to think some workaround
BTW, i suggest you to compare the debug UART log first

do you have an external PC connecting to the problematic module?
Will you consider to use AVMS FOTA upgrade ?

9

Since the rate of reproducibity is so low (<0.1%) , it is better to think some workaround

I agree, having a way to work around the issue would be sufficient. If I make a copy of the fwupdate app somewhere, do you think I could reuse it during a startup script to identify and recover from the issue (attempting to flash again)?

do you have an external PC connecting to the problematic module?

Yes, we have retrieved the module. I have been comparing logs but there is no error during the update that indicates the problem (when comparing to a unit which updated successfully).

Will you consider to use AVMS FOTA upgrade ?

AVMS FOTA upgrade used to not be usable over ethernet. The FOTA Service was also forcing the route to use cellular connection, even if an ethernet connection was available. Those aspects were showstoppers for our application.

10

I don’t know if you can still run fwupdate when module is in this faulty status
Have you tried that?

11

I’ll run some tests, I don’t think it’s available anymore, but maybe I can make a copy somewhere that cannot be overwritten.

12

Is there a way to provoke this issue so I can test a workaround multiple times?

For example a way to delete the content of /mnt/legato?

13

this directory is read-only…

14

Yes, I know. Yet it has been erased on my unit during the fwupdate process.

Please, is there any way to replicate this result?

15

i only know you can stop the legato framework…

16

Thanks for your help.

Do you know how I can get support on this?

17

probably you need to contact distributor

18

Deleting the content of /mnt/flash can be done after remounting it with mount -o remount,rw /mnt/flash

EDIT: my mistake, it doesn’t work for /mnt/legato which is what I really want to clean up

19

Hello,

After further investigation, it appears that the problem comes from a bad block in the lefwkro mtd:
image
image

Is there a suggested way to deal with this issue if it happens on a deployed unit?

20

i don’t know if this will damage the module by formatting the partition.
Here is an example to format /dev/mtd17: