Rollback despite system being marked as good

Legato Application Framework
1

I’m having an issue where even after the system passes the probation period it rolls back at some point. I’m trying to determine why this is happening and under what conditions does the system roll back even after passing probation? I check ‘legato status’, which shows the system is marked ‘good’. Is there any way I can check the cause for the rollback? Your help is appreciated.

2

Some additional details after a few rounds of testing. I found that the system rolls back when I’m running the device off a battery and I let the battery drain completely. When the battery is charged again and the device is powered it goes through the rollback process, despite the system being in a ‘good’ state. Ideally in production we’re not going to let the battery drain out but I’m trying to understand what causes this behavior. Any help would be appreciated.

3

HI Ktanikel
Legato currently doesn’t detect hardware faults (e.g. loss of power), and so will treat any quick reboot as a software fault, and roll back to golden master if too many occur in a short period. That’s why you can observe this when the battery drain completely .
Some improvements should be added into next versions of Legato, to avoid spurious rollbacks, to identify if unexpected resets is due to hardware faults ( loss of power, temperature, low voltage…)
best regards,

4

Hi plu,

Thanks for the reply, that explains the behavior on our devices. As a follow up question, does this occur even when devMode is enabled? I have devMode enabled on our devices and I read somewhere that it bypasses some of these checks.

5

Hi ktanikel,
you’re right!
see Developer Mode - Legato Docs
one method is to decrease the value of the probation period,

for example to set to 30sec put into the .adef file something like that:

processes: {
envVars: {
LE_PROBATION_MS = 30000
}

best regards,

6

Hi @plu

You mentioned some improvements to Legato to avoid spurious rollbacks on loss of power. Checked the legato change logs but no mention of this fix. Do you know if this is improved in a released Legato version or if this is still a problem in Legato 19.04?

7

Maybe this commit:
https://github.com/legatoproject/legato-af/commit/081e52ea65e6ce5c205d8bff7c534c64677c1dd1

8

Looks promising! Will do some testing.

9

For info, during cyclic reset tests on WP7607-1 / legato 19.02, I was also facing some unexpected rollback despite for system being marked as good.

It appears that this issue was linked to a wrong handling of the “bootCount” parameter. It has been fixed in 19.04.

You can refer to the following commit for details.

https://github.com/legatoproject/legato-af/commit/77e531931957953add45420bb093496baf4d2c48

When using 19.04 or above, I confirm the rollback does not happen during my reset test when the system is marked as good

10

Hi there,

Despite the promising comments about this being fixed in 19.04, I have been testing with the recent 21.05 release and I find that the rollback still happens after successive power cycles, regardless of whether the system is marked good.

Is there a way to disable this rollback behavior? It’s a major issue for us.

Thanks!

11

You can build your app to legato.cwe

12

Hi,

Did you manage to find a way to disable the rollback behaviour? I’m running legato application framework 19.11.6 and even though I mark the system as “good”, rollbacks sometimes occur.

13

have you tried to integrate your application to the legato.cwe?

14

I initially had my application baked into a “system” along with legato.cwe, which prevented the rollback issue (i think) presumably because if a rollback did occur, it would simply rollback to itself. However due to the limited flash size on the WP76 modem, my application became too large to be baked into a “system”, causing “fwupdate” to silently fail.

As such I have no choice but to have a separate “app” from the “system” and I am still experiencing rollbacks even though I run update -g after installing the “app”. Is there no way to completely disable rollbacks?

15

here says you can install the devMode app:

You might need to mark system good (update -g) after install devMode app

Another option is to contact distributor to request for a tool to adjust the Legato partition size in order to let you install a bigger legato.cwe

16

Thanks for the swift reply.

Sadly installing devMode isn’t a solution for me as it prevents the modem from going to sleep during inactivity.

I’ll contact my distributor regarding that tool, although I have already sold units to customers and it may not be achievable to modify their units.

In my attempt to disable rollbacks I have even modified the Legato Application Framework source code to remove what appears to be the rollback logic, yet rollbacks still occasionally occur, which is a critical issue for us. Is there no way to disable rollbacks in 100% of cases?

17

I guess the rollback is in here ( function CheckAndInstallCurrentSystem()):

18

Yes this is the file that I already modify in an attempt to disable rollbacks. I basically just remove the lines shown here:

Even after removing those lines, rollbacks seem to occur

19

How about removing lines from 2024 to 2105?

@cchenry , any idea how to disable the rollback feature?

20

I tried removing lines 2050 to 2054 but that caused errors when installing my legato “system”.
To clarify, I have a non-default “system” (.cwe) that I combine with the base legato system (.cwe) into one .spk file. My non-default “system” enables certain legato APIs, which I use in my “application” (.update file)