AT+WDSS throws error

1

I have installed the official leaf package for Legato Release 15 on my FX30s

9999999_9908088_SWI9X06Y_02.22.12.00_00_VERIZON_001.042_002-full.spk

I have a verizon sim in the unit. When I try to configure AirVantage services I am getting an error

I am not able to run AT+WDSC or AT+WDSS

Feb 23 23:06:46 fx30s user.notice root: run_getty.sh: No UART mapped to CONSOLE
Feb 23 23:06:47 fx30s cron.info crond[657]: time disparity of 27953048 minutes detected
Feb 23 23:06:48 fx30s user.notice root: config_avms_at: AT+WDSC? returned ERROR; retrying…
Feb 23 23:06:48 fx30s user.debug Legato: DBUG | atQmiLinker[1287]/atQmiLinker T=unknown | atForward.c QmiAtCmdCb() 161 | Received:at+WDSC?^M ' Feb 23 23:06:48 fx30s user.debug Legato: DBUG | atQmiLinker[1287]/atQmiLinker T=Writer | atLinker.c Writer() 334 | cmd: at+WDSC?^M ’
Feb 23 23:06:48 fx30s user.err Legato: =ERR= | atServerDaemon[1296]/atServer T=main | le_atServer.c ParseAtCmd() 2439 | Error in parsing AT command, lastState 0, current state 2
Feb 23 23:06:48 fx30s user.debug Legato: DBUG | atQmiLinker[1287]/atQmiLinker T=Reader | atForward.c SendIntermediateResponse() 416 | ##^M ERROR^M ##
Feb 23 23:06:51 fx30s user.notice root: config_avms_at: AT+WDSC? returned ERROR; retrying…
Feb 23 23:06:51 fx30s user.debug Legato: DBUG | atQmiLinker[1287]/atQmiLinker T=unknown | atForward.c QmiAtCmdCb() 161 | Received:at+WDSC?^M ' Feb 23 23:06:51 fx30s user.debug Legato: DBUG | atQmiLinker[1287]/atQmiLinker T=Writer | atLinker.c Writer() 334 | cmd: at+WDSC?^M ’
Feb 23 23:06:51 fx30s user.err Legato: =ERR= | atServerDaemon[1296]/atServer T=main | le_atServer.c ParseAtCmd() 2439 | Error in parsing AT command, lastState 0, current state 2
Feb 23 23:06:51 fx30s user.debug Legato: DBUG | atQmiLinker[1287]/atQmiLinker T=Reader | atForward.c SendIntermediateResponse() 416 | ##^M ERROR^M ##

2

how about this official release?
https://source.sierrawireless.com/resources/airlink/software_downloads/fx30-firmware/fx30-firmware-catm/#sthash.9iSahUIy.dpbs

3

Hi Jyijyi, Sunil.

We may also having a very similar issue with the R15 on FX30S. We are seeing AT+WDSC? report an error in both microcom /dev/ttyAT as well as in the logread logs from the at service startup.

Even worse, In our case, the avcService is reporting this error and then, within 5 minutes, causing a watchdog reboot of the entire modem. Stopping the app doesnt prevent the reboot so we are at a standstill (for now) moving to R15. I suspect it related to the WDSC issue, but couldnt be certain.

@Jyijyi I followed your link to the official release but it doesnt have any R15 content, just R14 and earlier. https://source.sierrawireless.com/resources/airlink/software_downloads/fx30-firmware/fx30-firmware-catm/#sthash.9iSahUIy.dpbs

We loaded the R15 legato via leaf and we got the R15 legato cwe firmware file from Sierra Wireless in Australia (courtesy of Bill Z) with the RS232 DCD control signal memory leak fix for ioctl() included.

Would you be able to provide a summary or link to the details of this R15 bug/issue for either avcService or AT+WDSC? so we can apply the same fix to the R15 code base we are using?

Much appreciated
Brett

WDSC Clues:
Mar 17 13:18:38 fx30s user.notice root: config_avms_at: AT+WDSC? returned ERROR; retrying…
Mar 17 13:18:38 fx30s user.debug Legato: DBUG | atQmiLinker[1291]/atQmiLinker T=unknown | atForward.c QmiAtCmdCb() 161 | Received:at+WDSC?^M ' Mar 17 13:18:38 fx30s user.debug Legato: DBUG | atQmiLinker[1291]/atQmiLinker T=Writer | atLinker.c Writer() 334 | cmd: at+WDSC?^M ’
Mar 17 13:18:38 fx30s user.err Legato: =ERR= | atServerDaemon[1300]/atServer T=main | le_atServer.c ParseAtCmd() 2439 | Error in parsing AT command, lastState 0, current state 2
Mar 17 13:18:38 fx30s user.debug Legato: DBUG | atQmiLinker[1291]/atQmiLinker T=Reader | atForward.c SendIntermediateResponse() 416 | ##^M ERROR^M ##
Mar 17 13:18:39 fx30s user.warn Legato: -WRN- | modemDaemon[1359]/le_pa T=main | pa_sim_qmi.c MapSimState() 1012 | More than one application 2

…

The Watchdog issue after 5 minutes (My real issue)
Mar 17 13:23:25 fx30s user.crit Legato: CRT | watchdog[1270]/watchdogDaemon T=main | watchdog.c WatchdogHandleExpiry() 487 | Mandatory watchdog double fault on process [avcService][avcDaemon]
Mar 17 13:23:25 fx30s user.emerg Legato: EMR | watchdog[1270]/le_pa_wdog T=main | pa_wdog.c pa_wdog_Shutdown() 72 | Watchdog expired. Restart device.
Mar 17 13:23:25 fx30s user.emerg Legato: EMR | xtensorMain[2134]/framework T=main | messagingSession.c ClientSocketHangUp() 870 | Session closed by server (xtensorMain.xtensorMain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Mar 17 13:23:25 fx30s user.emerg Legato: EMR | dcsDaemon[1334]/framework T=main | messagingSession.c ClientSocketHangUp() 870 | Session closed by server (le_wdog:572ba176b790548eed89b7ba6f9f4b14).
Mar 17 13:23:25 fx30s user.emerg Legato: EMR | cellNetService[1325]/framework T=main | messagingSession.c ClientSocketHangUp() 870 | Session closed by server (cellNetService.watchdogChain.le_wdog:572ba176b790548eed89b7ba6f9f4b14).

4

I don’t see error on AT+WDSC?, are we using same FW version?


ati3
Manufacturer: Sierra Wireless, Incorporated
Model: WP7702
Revision: SWI9X06Y_02.36.08.09 c094f4 jenkins 2022/06/24 03:14:28
IMEI: 352653090612345
IMEI SV:  8
FSN: VU213430812345
+GCAP: +CGSM

OK
ati8
Legato Ver: 21.05.0.5145594a_276fe72bcc9e592e0666ed13a3023975
Yocto Ver:  SWI9X06Y_03.00.07.00 2022-06-27_10:01:05
OS Ver: Linux version 4.14.253 (oe-user@oe-host) ( gcc version 9.3.0 (GCC), GNU ld (GNU Binutils) 2.34.0.20200220) #1 PREEMPT Mon Jun 27 07:55:42 UTC 2022
LK Ver: 1.3.0_1aea4d8ec8
RootFS Ver: SWI9X06Y_03.00.07.00 2022-06-27_10:01:05
UserFS Ver: unknown
MCU Ver: 002.015

OK
at+wdsc?
+WDSC: 0,0
+WDSC: 1,0
+WDSC: 2,0
+WDSC: 3,60
+WDSC: 4,15,60,240,480,1440,2880
+WDSC: 5,0
+WDSC: 6,0

OK
5

Not quite the same version. The build we are using came from Bill Z at Sierra Wireless in Australia. He added a patch for us for the serial DCD memory leak issue and he is also not seeing the reboot. Looking at the versions (below), it looks like you may have a point release better version of the Yocto and RootFS. I will send Bill the link to this chat. my WDSC? command simply fails with ERROR. If I have an older code base, it may also explain the rebooting Legato. The firmware came from Bill’s build and the Dev environment came from Leaf within the last week or two.
leaf status
│ Workspace: /home/dtluser/myWorkspace │
│ Profile: R15fx30catmstable [current] (sync) │
│ Included │ swi-fx30-catm_4.1.0 │ SDK for FX30-CATM (Release 15.0.5.002 + Legato 21.05.0) │

root@fx30s:~# microcom /dev/ttyAT
ati3
Manufacturer: Sierra Wireless, Incorporated
Model: FX30S(WP7702)
Revision: SWI9X06Y_02.36.08.02 c3f5ef jenkins 2021/09/07 08:00:51
IMEI: 354723090157826
IMEI SV: 6
FSN: VU011585670510
+GCAP: +CGSM

OK
ati8
Legato Ver: 21.05.0.65e00130_ee790a73c4227bb8ae4b393065f41da2_modified
Yocto Ver: SWI9X06Y_03.00.06.00 2023-02-06_01:39:56
OS Ver: Linux version 4.14.253 (oe-user@oe-host) ( gcc version 9.3.0 (GCC), GNU ld (GNU Binutils) 2.34.0.20200220) #1 PREEMPT Tue Jul 26 06:51:15 UTC 2022
LK Ver: 1.3.0_1aea4d8ec8
RootFS Ver: SWI9X06Y_03.00.06.00 2023-02-06_01:39:56
UserFS Ver: unknown
MCU Ver: 002.015

OK
at+wdsc?
ERROR

6

ps.
A quick thought from Bill on this topic, potentially relevant. We are testing this upgrade on an ex Octave FX30S. (Only stock available was Octave) which we have repurposed for non Octave use - we dont use Octave.

Bill said:
I am not sure a firmware upgrade can fully convert an Octave FX30S to standard FX30S.

There is a discussion here which explains at least it would have issue with connecting to AV. Possible this is the cause of triggering avcService watchdog.

Bill said: AT+WDSC? works on my FX30S.

Brett aid (me): We didnt have any issues replacing Octave using R14.

7

seems the modem version is different from mine, i am using SWI9X06Y_02.36.08.09

8

Hi Jyijyi,

The radio firmware version should not be the issue here. Brett is using the radio version certified to Telstra which is a little different than the one in your WP7702.

I run the same firmware as Brett’s on a standard FX30S(non-Octave) and I cannot replicate this issue. So far I can see the difference between Brett and my side is that Brett is using an Octave FX30S. I tend to think it’s Octave version of FX30S related, but Brett also mentioned no issue to run R14 on the same device. Would be useful to get some input from Octave team on how to apply standard firmware to Octave FX30 I suppose? Otherwise, best chance is to go with standard FX30S I’d think.

9

is that application list in “app status” different between yours and Brett’s module?
I wonder if some application is blocking the AT+WDSC?

10

Full app status and app info list below if it helps. the udpsend, xtensorAutoExec and xtensorMain ones are ours and I have reproduced this issue with them removed. Happy to delete again and retry if this is important.

root@fx30s:~# app status
[running] atAirVantage
[running] atQmiLinker
[running] atService
[running] audioService
[running] avcService
[running] cellNetService
[running] dataConnectionService
[running] fwupdateService
[running] gpioService
[running] modemService
[running] portService
[running] positioningService
[running] powerMgr
[running] powerService
[running] qmiAirVantage
[running] secStore
[stopped] smsInboxService
[stopped] spiService
[stopped] tools
[stopped] voiceCallService
[stopped] wifi
[stopped] wifiApTest
[stopped] wifiClientTest
[stopped] wifiWebAp
[running] columbiaAtService
[running] factoryRecoveryService
[stopped] udpsend
[stopped] xtensorAutoExec
[running] xtensorMain
root@fx30s:~# app info
atAirVantage
status: running
running processes:
atAirVantage[1283] (1283)
app.name: atAirVantage
app.md5: 891ad81bcd93cacfc1a0c9d65d553106
app.version:
legato.version: 21.05.0.65e00130

atQmiLinker
status: running
running processes:
atQmiLinker[1291] (1291, 1546, 1547, 1548, 1550, 1551)
app.name: atQmiLinker
app.md5: 4907efc30232eea476cdfefdff3ffb70
app.version:
legato.version: 21.05.0.65e00130

atService
status: running
running processes:
atServerDaemon[1300] (1300)
atClientDaemon[1299] (1299)
app.name: atService
app.md5: 3f3c60d0c9b04ecd5887d68ba3e3abf8
app.version:
legato.version: 21.05.0.65e00130

audioService
status: running
running processes:
audioDaemon[1308] (1308, 1504, 1505, 1506, 1507, 1509)
app.name: audioService
app.md5: 56c2ca78a36c4d64b3d9edab4a8dbf4d
app.version:
legato.version: 21.05.0.65e00130

avcService
status: running
running processes:
avcDaemon[1317] (1317)
app.name: avcService
app.md5: c95bd4e16e30567f0bec7cf28bee06ae
app.version:
legato.version: 21.05.0.65e00130

cellNetService
status: running
running processes:
cellNetService[1325] (1325)
app.name: cellNetService
app.md5: e1830bd9be2aa31d6fa86ddbd7379c8b
app.version:
legato.version: 21.05.0.65e00130

dataConnectionService
status: running
running processes:
dcsDaemon[1333] (1333)
app.name: dataConnectionService
app.md5: c002cee3b318e1498c4a9af8541e5896
app.version:
legato.version: 21.05.0.65e00130

fwupdateService
status: running
running processes:
fwupdateDaemon[1342] (1342, 1569, 1570, 1571, 1575)
app.name: fwupdateService
app.md5: aaabad1475a2334fbb2845753eb41db7
app.version:
legato.version: 21.05.0.65e00130

gpioService
status: running
running processes:
gpioService[1351] (1351)
app.name: gpioService
app.md5: d39b08a49107858296fcbbe031bcae84
app.version:
legato.version: 21.05.0.65e00130

modemService
status: running
running processes:
modemDaemon[1359] (1359, 1586, 1587, 1589, 1590, 1591, 1593, 1594, 1596, 1597, 1601, 1602, 1604, 1605, 1607, 1609, 1614, 1615, 1619, 1621, 2143)
rSimDaemon[1360] (1360, 1543, 1545, 1549)
app.name: modemService
app.md5: b3a2c02d07372053becca70cdad9c01f
app.version:
legato.version: 21.05.0.65e00130

portService
status: running
running processes:
portDaemon[1368] (1368)
app.name: portService
app.md5: 4f588a547ccc1e0a6ac5144e090be7cb
app.version:
legato.version: 21.05.0.65e00130

positioningService
status: running
running processes:
posDaemon[1376] (1376, 1535, 1537, 1539, 1541, 1544)
app.name: positioningService
app.md5: 674aba983f38835304b9e5d3da1a6e51
app.version:
legato.version: 21.05.0.65e00130

powerMgr
status: running
running processes:
powerManagerService[1387] (1387)
app.name: powerMgr
app.md5: 806ed06ad4af06c5baac2a466b66e671
app.version:
legato.version: 21.05.0.65e00130

powerService
status: running
running processes:
powerDaemon[1395] (1395)
app.name: powerService
app.md5: 3ef4411fb0e5c77f2716125172962866
app.version:
legato.version: 21.05.0.65e00130

qmiAirVantage
status: running
running processes:
qmiAirVantage[1403] (1403)
app.name: qmiAirVantage
app.md5: d2ba840971b920d75b0b203f6e4a5ce4
app.version:
legato.version: 21.05.0.65e00130

secStore
status: running
running processes:
secStore[1412] (1412, 1536, 1538, 1540, 1542)
app.name: secStore
app.md5: 7a8f3dca929c85a92c71c7739bc87b65
app.version:
legato.version: 21.05.0.65e00130

smsInboxService
status: stopped
app.name: smsInboxService
app.md5: c95b4ebc99d92da819b9db86f26d3686
app.version:
legato.version: 21.05.0.65e00130

spiService
status: stopped
app.name: spiService
app.md5: 17bea243f605c2c6f509d31a17b98231
app.version: 0.1.0
legato.version: 21.05.0.65e00130

tools
status: stopped
app.name: tools
app.md5: 467b33cb06a6f4614da6562ac6cc7ddc
app.version:
legato.version: 21.05.0.65e00130

voiceCallService
status: stopped
app.name: voiceCallService
app.md5: efe2c93d22c0f5b27fbcc638ee04f143
app.version:
legato.version: 21.05.0.65e00130

wifi
status: stopped
app.name: wifi
app.md5: 429a2bbf0cf59d56a361bfdb18d44f29
app.version: 21.05.0
legato.version: 21.05.0.65e00130

wifiApTest
status: stopped
app.name: wifiApTest
app.md5: 5e580f7bc3123b7bd6917298d936bc45
app.version: 21.05.0
legato.version: 21.05.0.65e00130

wifiClientTest
status: stopped
app.name: wifiClientTest
app.md5: 17fb780ceae37cbc495fd03703e3d594
app.version: 21.05.0
legato.version: 21.05.0.65e00130

wifiWebAp
status: stopped
app.name: wifiWebAp
app.md5: 3212586638bfe12a742c9a848225e0a3
app.version:
legato.version: 21.05.0.65e00130

columbiaAtService
status: running
running processes:
columbiaAtService[1421] (1421)
app.name: columbiaAtService
app.md5: aec441e556b5624226b853fe5e4ae3e2
app.version:
legato.version: 21.05.0.62a8f41c

factoryRecoveryService
status: running
running processes:
factoryRecoveryService[1431] (1431)
app.name: factoryRecoveryService
app.md5: d93fcdb531b6497029c7d29098e67f56
app.version:
legato.version: 21.05.0.62a8f41c

udpsend
status: stopped
app.name: udpsend
app.md5: 1b0c744067785d810c33f7c97d521505
app.version: U1000.202303101412
legato.version: 21.05.0

xtensorAutoExec
status: stopped
app.name: xtensorAutoExec
app.md5: 1eec08c61d1f68186aff5e15b09963d1
app.version: A1001.202303101412
legato.version: 21.05.0

xtensorMain
status: running
running processes:
xtensorMain[2131] (2131, 2136)
app.name: xtensorMain
app.md5: 37665b4a8094f73e8647e2f71a441d93
app.version: M1004c.202303161616
legato.version: 21.05.0

root@fx30s:~#

11

I was using the Verizon release directly off leaf and I could never get it to work. What was interesting is that when I built the Yocto image off the source code and updated my FX30s it worked properly and I could make an AV connection but the ones packaged with leaf did not work for me.

12

Hi @billzhou
Probably you can make a post here to interact with Octave team:

It seems this is not image problem but only related to the hardware

13

POTENTIAL SOLUTION

The issue was that our Octave modems, once upgraded to R15 started rebooting after 5-8 minutes. It seems that this is because AT+WDSC? and related commands started failing (ERROR), which in turn resulted in AVC/QMI failing to initialise and causing a modem wide watchdog reboot.

The issue doesn’t occur when you take a non-Octave generic modem and then apply R15. Ie. AT+WDSC? works and no reboot occurs. We have a number of Octave modems that were provided due to low stock levels for FX30S.

The good news is that we have found that downgrading an Octave modem to R14 and then upgrading it to R15 solves the problem and the AT+WDSC? command starts working and the reboots go away. I haven’t completed testing so while I cant be 100% certain it is stable – it seems ok.

This suggests that the R15 upgrade doesn’t replace all of the components in the modem, and R14 may replace the missing Octave specific settings. This appears to be an Octave related issue and we now have a workaround.

A painful week, but with a positive end for the weekend.