When WP7605 is restarted, the modemDemo disappears

The following URLs and beyond are also under development.
https://forum.legato.io/t/cdc-ecm-data-connection-failing/6825

Currently, we are trying to get the WP7605 with the modemDemo app installed to work on a board developed in-house.
However, CDC-ECM does not seem to be working well and ping 8.8.8.8 results in Network Unreachable.

To investigate, we set the host CPU on our own development board to disable and connected with WP7605 and a Linux PC via USB cable.
After connecting via SSH and checking with the “app status” command, we found that the modemDemo application, which we thought we had installed, had disappeared.

After investigation, we found that the modemDemo app disappears when the following steps are performed.

  1. Install the modemDemo app on WP7605, configure various settings, and confirm that ping 8.8.8.8 passes
  2. Unplug the USB cable from the Linux PC
  3. Turn off the power of the in-house development board (WP7605 is also turned off).
  4. Turn on the in-house development board (WP76905 will also boot up)
  5. Connect the USB cable to the Linux PC
  6. “app status” via SSH shows that the modemDemo app is gone.

Linux PC is Panasonic CF-SZ6, OS is Ubuntu 18.04.06 LTS.

There are some patterns in which the modemDemo app remains without disappearing, and the following is an example.

  • If you follow the above procedure and only power On/Off the in-house development board without plugging and unplugging the USB cables No.2 and No.5, the modemDemo app will not disappear.
  • If you change the Linux PC, the modemDemo app will remain without disappearing even if you do the same thing as above
    If the Linux PC is Thinkpad X220 and the OS is xUbuntu 20.04 LTS, the modemDemo app does not disappear
    When the Linux PC is Thinkpad X230 and OS is Ubuntu 18.04.06 LTS, the modemDemo app does not disappear

The following URL was referenced and it stated that logread logs are required, so I am attaching them.
https://forum.legato.io/t/legato-rolling-back-config-tree-and-app/2745/4

The log of normal system is
logread(Normal).txt (376.2 KB)

The log when the modemDemo app disappears is
logreaad(disappear).txt (376.3 KB)

Please tell us why the modemDemo app disappears.

from the logread, seems there is crash:


Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   40.973719] WARNING: CPU: 0 PID: 996 at /usr/src/kernel/net/netfilter/nf_nat_core.c:393 nf_nat_setup_info+0x88/0x7c4()
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   40.973734] Modules linked in: cfg80211
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   40.973772] CPU: 0 PID: 996 Comm: kworker/u2:6 Not tainted 3.18.131 #1
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   40.973804] Workqueue: uether process_rx_w
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   40.973825] ------------[ cut here ]------------
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   40.977424] WARNING: CPU: 0 PID: 996 at /usr/src/kernel/kernel/locking/mutex.c:870 mutex_trylock+0x78/0x174()
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   40.987426] DEBUG_LOCKS_WARN_ON(in_interrupt())
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   40.991693] Modules linked in: cfg80211
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   40.995647] CPU: 0 PID: 996 Comm: kworker/u2:6 Not tainted 3.18.131 #1
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.002224] Workqueue: uether process_rx_w
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.006269] [<c0014428>] (unwind_backtrace) from [<c00122ac>] (show_stack+0x10/0x14)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.014019] [<c00122ac>] (show_stack) from [<c001f490>] (warn_slowpath_common+0x60/0x80)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.022110] [<c001f490>] (warn_slowpath_common) from [<c001f4dc>] (warn_slowpath_fmt+0x2c/0x3c)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.030769] [<c001f4dc>] (warn_slowpath_fmt) from [<c0790450>] (mutex_trylock+0x78/0x174)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.038890] [<c0790450>] (mutex_trylock) from [<c001acbc>] (sierra_smem_errdump_save_frame+0x20/0x124)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.048218] [<c001acbc>] (sierra_smem_errdump_save_frame) from [<c00144c0>] (unwind_backtrace+0x98/0x128)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.057766] [<c00144c0>] (unwind_backtrace) from [<c00122ac>] (show_stack+0x10/0x14)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.065490] [<c00122ac>] (show_stack) from [<c001f490>] (warn_slowpath_common+0x60/0x80)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.073559] [<c001f490>] (warn_slowpath_common) from [<c001f540>] (warn_slowpath_null+0x18/0x20)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.082334] [<c001f540>] (warn_slowpath_null) from [<c068201c>] (nf_nat_setup_info+0x88/0x7c4)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.090930] [<c068201c>] (nf_nat_setup_info) from [<c06e097c>] (nf_nat_masquerade_ipv4+0x13c/0x158)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.099913] [<c06e097c>] (nf_nat_masquerade_ipv4) from [<c06e3304>] (masquerade_tg+0x40/0x60)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.108475] [<c06e3304>] (masquerade_tg) from [<c06e14b4>] (ipt_do_table+0x694/0x790)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.116271] [<c06e14b4>] (ipt_do_table) from [<c06dd4b4>] (nf_nat_ipv4_fn+0x118/0x210)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.124171] [<c06dd4b4>] (nf_nat_ipv4_fn) from [<c06dd660>] (nf_nat_ipv4_out+0x3c/0xec)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.132152] [<c06dd660>] (nf_nat_ipv4_out) from [<c06640f4>] (nf_iterate+0x68/0xb4)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.139795] [<c06640f4>] (nf_iterate) from [<c066419c>] (nf_hook_slow+0x5c/0x110)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.147265] [<c066419c>] (nf_hook_slow) from [<c0698700>] (ip_output+0xb8/0xd4)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.154561] [<c0698700>] (ip_output) from [<c063e080>] (__netif_receive_skb_core+0x400/0x50c)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.163066] [<c063e080>] (__netif_receive_skb_core) from [<c0640118>] (process_backlog+0x70/0x110)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.172003] [<c0640118>] (process_backlog) from [<c063fa60>] (net_rx_action+0x80/0x158)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.179946] [<c063fa60>] (net_rx_action) from [<c0021f20>] (__do_softirq+0x180/0x1e8)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.187797] [<c0021f20>] (__do_softirq) from [<c002205c>] (do_softirq+0x44/0x54)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.195173] [<c002205c>] (do_softirq) from [<c0640658>] (netif_rx_ni+0x34/0x70)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.202471] [<c0640658>] (netif_rx_ni) from [<c04372cc>] (process_rx_w+0x170/0x1d4)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.210109] [<c04372cc>] (process_rx_w) from [<c0031014>] (process_one_work+0x1bc/0x2e0)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.218138] [<c0031014>] (process_one_work) from [<c0031dd0>] (worker_thread+0x2dc/0x3b8)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.226337] [<c0031dd0>] (worker_thread) from [<c003567c>] (kthread+0xc8/0xd4)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.233578] [<c003567c>] (kthread) from [<c000e798>] (ret_from_fork+0x14/0x3c)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.240741] ---[ end trace 9aeba0951e9fb07b ]---
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.245313] [<c0014428>] (unwind_backtrace) from [<c00122ac>] (show_stack+0x10/0x14)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.253077] [<c00122ac>] (show_stack) from [<c001f490>] (warn_slowpath_common+0x60/0x80)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.261155] [<c001f490>] (warn_slowpath_common) from [<c001f540>] (warn_slowpath_null+0x18/0x20)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.269873] [<c001f540>] (warn_slowpath_null) from [<c068201c>] (nf_nat_setup_info+0x88/0x7c4)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.278512] [<c068201c>] (nf_nat_setup_info) from [<c06e097c>] (nf_nat_masquerade_ipv4+0x13c/0x158)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.287554] [<c06e097c>] (nf_nat_masquerade_ipv4) from [<c06e3304>] (masquerade_tg+0x40/0x60)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.296045] [<c06e3304>] (masquerade_tg) from [<c06e14b4>] (ipt_do_table+0x694/0x790)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.303856] [<c06e14b4>] (ipt_do_table) from [<c06dd4b4>] (nf_nat_ipv4_fn+0x118/0x210)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.311755] [<c06dd4b4>] (nf_nat_ipv4_fn) from [<c06dd660>] (nf_nat_ipv4_out+0x3c/0xec)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.319697] [<c06dd660>] (nf_nat_ipv4_out) from [<c06640f4>] (nf_iterate+0x68/0xb4)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.327382] [<c06640f4>] (nf_iterate) from [<c066419c>] (nf_hook_slow+0x5c/0x110)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.334845] [<c066419c>] (nf_hook_slow) from [<c0698700>] (ip_output+0xb8/0xd4)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.342142] [<c0698700>] (ip_output) from [<c063e080>] (__netif_receive_skb_core+0x400/0x50c)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.350650] [<c063e080>] (__netif_receive_skb_core) from [<c0640118>] (process_backlog+0x70/0x110)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.359544] [<c0640118>] (process_backlog) from [<c063fa60>] (net_rx_action+0x80/0x158)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.367576] [<c063fa60>] (net_rx_action) from [<c0021f20>] (__do_softirq+0x180/0x1e8)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.375383] [<c0021f20>] (__do_softirq) from [<c002205c>] (do_softirq+0x44/0x54)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.382759] [<c002205c>] (do_softirq) from [<c0640658>] (netif_rx_ni+0x34/0x70)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.390015] [<c0640658>] (netif_rx_ni) from [<c04372cc>] (process_rx_w+0x170/0x1d4)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.397689] [<c04372cc>] (process_rx_w) from [<c0031014>] (process_one_work+0x1bc/0x2e0)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.405768] [<c0031014>] (process_one_work) from [<c0031dd0>] (worker_thread+0x2dc/0x3b8)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.413935] [<c0031dd0>] (worker_thread) from [<c003567c>] (kthread+0xc8/0xd4)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.421126] [<c003567c>] (kthread) from [<c000e798>] (ret_from_fork+0x14/0x3c)
Jan 21 12:40:07 swi-mdm9x28-wp user.warn kernel: [   41.428282] ---[ end trace 9aeba0951e9fb07c ]---
Jan 21 12:40:46 swi-mdm9x28-wp user.info Legato:  INFO | supervisor[835]/supervisor T=main | supervisor.c HandleRebootExpiry() 534 | Expired reboot timer
Jan 21 12:41:27 swi-mdm9x28-wp user.info kernel: [  121.441244] sierra_startup_monitor
Jan 21 12:43:34 swi-mdm9x28-wp authpriv.info dropbear[2046]: Child connection from 192.168.2.3:53234
Jan 21 12:43:35 swi-mdm9x28-wp authpriv.notice dropbear[2046]: Auth succeeded with blank password for 'root' from 192.168.2.3:53234
Jan 21 12:43:35 swi-mdm9x28-wp authpriv.warn dropbear[2047]: lastlog_perform_login: Couldn't stat /var/log/lastlog: No such file or directory
Jan 21 12:43:35 swi-mdm9x28-wp authpriv.warn dropbear[2047]: lastlog_openseek: /var/log/lastlog is not a file or directory!

BTW, have you marked the system to be good by “update -g” after installing the modemdemo application?

Lastly, you can see here to modify wifi.sdef in order to embed your application in the legato image:

Thank you very much for your reply.
I have made some progress thanks to your advice.

I ran “update modemDemo.wp76xx.update 192.168.2.2 -g”.
Before I ran it, the modemDemo app was disappearing after one power on/off, but now it doesn’t disappear until 3 times (it disappears after the 4th time).
I am attaching the logread log of that case.
20250122_logread.txt (355.1 KB)

You mentioned that it is crashing, is the crash still the cause this time?
Also, are there any possible contributing factors to the crash?
If you know of any, please let us know.
For example, is there a problem with power management or USB control?
Or is the host CPU doing something?

One more question regarding the thread you linked to in one of your previous posts.
Flash size wp7607
I think you are asking me to add the modemDemo.adef to the wifi.sdef and create a legato.cwe file.
Could you please tell me why this is?
Is it for the following reasons?

  • Legato has a rollback function in case of a crash.
  • If the rollback is triggered by a crash, the modemDemo app will be lost.
  • If the modemDemo app is already in the original environment, the modemDemo will not be lost even if the rollback is caused by a crash.

If this guess is correct, I think this is a good idea, but I feel that if the cause of the crash is not resolved, it will cause problems later on.

did you also check “legato status” to see if the legato framework has marked to be good system?

I am not sure, but you can do a simple test.
For example, in the adef of your modemdemo, set “start: manual” to be manual start application. After you download the application to module, you can observe if there is any crash message.

Yes, that is correct!

But it is quite strange that it only happens to Panasonic CF-SZ6.
It seems to be not related to module problem.
Did you try this PC with a USB hub and see what happens?

Thanks for your advice.
I tried running “legato status”.
I got the following result.

Systems installed: 
  2 [good]
  3 [tried 1] <-- current
Legato framework is running.

The above is the result of the in-house development board, but when I checked with devkit, the result was as follows.

Systems installed:
  2 [good] <-- current
Legato framework is running.

I am concerned about [tried 1].
Is the -g option of “update -g” not working?

after installing application and entering “update -g”, the system should mark good.

Do you mean even you type “update -g”, it still shows [tried 1] ?

FYI, I don’t see problem with this command with my module in R17, please see below:


root@fx30:~# legato version
21.05.2.rc2_7f59219499ecf02274c4ac7575e4f8c5_modified
root@fx30:~# legato status
Systems installed:
  37 [good] <-- current
Legato framework is running.
root@fx30:~# app remove modemDemo1
Removing app 'modemDemo1'...

root@fx30:~#
root@fx30:~# legato status
Systems installed:
  37 [good]
  38 [tried 1] <-- current
Legato framework is running.
root@fx30:~# update /tmp/modemDemo1_R17.wp76xx.update
Unpacking package: 100% ++++++++++++++++++++++++++++++++++++++++++++++++++
Applying update: 100% ++++++++++++++++++++++++++++++++++++++++++++++++++
SUCCESS
root@fx30:~# legato status
Systems installed:
  37 [good]
  39 [tried 1] <-- current
Legato framework is running.
root@fx30:~# app status
[running] atAirVantage
[running] atQmiLinker
[running] atService
[running] audioService
[running] avcService
[running] cellNetService
[running] dataConnectionService
[running] fwupdateService
[running] gpioService
[running] modemService
[running] portService
[running] positioningService
[running] powerMgr
[running] powerService
[running] qmiAirVantage
[running] secStore
[stopped] smsInboxService
[stopped] spiService
[stopped] tools
[stopped] voiceCallService
[stopped] wifi
[stopped] wifiApTest
[stopped] wifiClientTest
[running] wifiService
[stopped] wifiWebAp
[stopped] UART_TCP_Demo1
[running] hello1
[running] modemDemo1
root@fx30:~# update -g
System is now marked 'Good'.
root@fx30:~# legato status
Systems installed:
  39 [good] <-- current
Legato framework is running.
  3 [tried 1] <-- current

The reason for the above is that I made a mistake.
“update modemDemo.wp76xx.update 192.168.2.2 -g” did not enable the -g option.
I followed the same steps as in your post and [tried 1] disappeared.
In this state, I can now power on and off up to 5 times without the modemDemo app disappearing.
A little progress.

I also tried the USB hub, but the modemDemo disappeared.

The best solution to this problem is to figure out why it crashes, so that is something I would like to investigate.
I would like to proceed with the creation of the legato.cwe file you suggested as the next best thing.

So, investigating the crash, could you please tell me the steps you suggested in your earlier post?

  1. download the modemDemo (update modemDemo.wp76xx.update 192.168.2.2)
  2. run “update -g”
  3. run “app status”
  4. run “app start modemDemo”
  5. turn the power off and on
    Repeat No.3 to 5 until no more “modemDemo” appears in “app status”.

Is this procedure correct?
If a crash occurs, where can I find the crash message?
Is it logread?

you can find the crash message in UART2 which is the linux console port.

BTW, does the crash related to the sharing CDC-ECM data operation here?

BTW, does the crash related to the sharing CDC-ECM data operation here?

Are you saying that you believe the crash is a hardware factor?
If so, are you saying that this is not what this Forum deals with?
If so, I agree with you.
I think this is a matter to be dealt with elsewhere.

Thanks for your advice on UART2.

I will also try to create a legato.cwe file.
If any questions arise here, please let me ask again.

I don’t know what the crash relates.
It is quite strange it only happens to particular PC (i.e. Panasonic CF-SZ6), but not in all other PC.
You might need to take more test to see if it only happens to particular Panasonic CF-SZ6 or all Panasonic CF-SZ6.
In addition, you might need to investigate if no CDC-ECM data sharing, will the problem happen?
Besides, setting adef of your modemdemo is just to see if manual start of application will cause the issue or not.
This might help to understand what is causing the crash.

Thank you for your advice.

I am working on building the Legato.cwe file in parallel with investigating the cause of the crash.

I built it with Ubuntu 18.04.06 LTS and got the same error as Make wp76xx error.
I was able to work around this by installing the following package.

sudo apt-get install libtool
sudo apt-get install cmake
sudo apt-get install autoconf-archive
sudo apt-get install pkg-config
sudo apt-get install flex
sudo apt-get install bison
sudo apt-get install byacc
sudo apt-get install gperf
sudo apt-get install libncurses-dev

I have confirmed that I can build normally, but when I change the wifi.sdef as you advised as follows, I get the error.

/home/systemac/.leaf/wp76-legato_19.02.0-201903211732/apps/sample/modemDemo/demoComponent/Component.cdef:17:15: error: Couldn't find file 'send_util.api'.
Makefile:605: recipe for target 'system' failed

Could you please tell me how to resolve this error?

here says you need to add the path “${LEGATO_ROOT}/apps/sample/modemDemo” to “interfacesearch” section in sdef file:

Thank you for your advice.
I added the “interfacesearch” section as you suggested and the build went through.

I updated the built legato.cwe to WP7605 and now the modemDemo is not disappearing.
However, 4 in 30 times, it does not respond to the “app status” command, let alone ping.
It did respond to logread, so I am attaching the log of that case.
logread1.txt (136.9 KB)
Is this also caused by the crash?

Also, could you please answer the following question about the cwe file?

  • Is there any way to know how many crashes or rollbacks have occurred?
  • Is legato.cwe the Base Legato System (.cwe)?
  • Is it popular to build cwe files?
    So far I have only dealt with Device Package (.spk) files and Widnows EXE files.
    If I needed some application, I used Overlay Legato System (.update).
    I am still concerned if legato.cwe is building correctly. (I’m wondering if there are any compile options, etc.)
  • Is it correct to assume that the legato.cwe built in the following environment is the same as the one in the Device Package(.spk)?
    1.The $LEGATO_ROOT in the environment created by the “leaf setup” command.
    2.The environment downloaded from “Base Legato System: 19.02.0: Download"
    I confirmed that there is no difference between $LEGATO_ROOT and “Base Legato System: 19.02.0: Download” by Diff.
  • If we update the firmware in the Device Package(.spk) file in the future, will the legato.cwe (Base Legato System(.cwe)) that we modified this time also be updated and the modemDemo will disappear?
    Currently, WP7605 is using firmware Release 13 and may be upgraded to Release 17 in the future.

did you check “legato status” when problem happens?

you can run a script to check AT!GSTATUS? which will return “current time”
After reset, this value should be reset to zero

yes, it is.

Yes, some customer also builds their application to legaot.cwe

“legato version” returns different values as they are built by different machine

Yes, it will be overwritten.
You can simply do a test to upgrade to R16.3