DataHub - API calls high CPU usage

tomalex · November 30, 2020, 7:24pm

Hi all,

We have also started to use the datahub application in legato, one problem we have noticed that under frequent updating to the datahub (around 200 calls per second) we have around 20% CPU usage in the client application and 20% load in the datahub. It seems the problem is related to the interface. Legato framework version is 20.04. Is it a known limitation? Is there any suggestion for improvements? We have not investigate deeply yet, we thought maybe we could ask in forum first.

Thanks,
tom

jyijyi · December 1, 2020, 9:00am

how to reproduce the issue step by step?
I tried on WP76 with the mangoh source, I don’t see 20% cpu load:

root@swi-mdm9x28-wp:~# app status | grep data
[running] dataConnectionService
[running] dataHub
root@swi-mdm9x28-wp:~#
root@swi-mdm9x28-wp:~# top

top - 16:58:52 up 9 min,  1 user,  load average: 2.28, 1.95, 1.06
Tasks: 197 total,   1 running, 196 sleeping,   0 stopped,   0 zombie
top - 16:58:55 up 9 min,  1 user,  load average: 2.28, 1.95, 1.06
Tasks: 197 total,   1 running, 196 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.7 us,  6.6 sy,  0.0 ni, 90.4 id,  1.3 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :   158792 total,     6452 free,    75924 used,    76416 buff/cache
KiB Swap:        0 total,        0 free,        0 used.    65332 avail Mem

  137 root      rt   0       0      0      0 S  4.5  0.0   0:01.44 cfinteracti+
 3378 root      20   0    3004   1256    880 R  2.6  0.8   0:00.23 top
    7 root      20   0       0      0      0 S  0.3  0.0   0:00.47 rcu_preempt
   22 root      20   0       0      0      0 S  0.3  0.0   0:01.87 kworker/u2:2
  137 root      rt   0       0      0      0 S  0.3  0.0   0:01.45 cfinteracti+
  139 root      20   0       0      0      0 S  0.3  0.0   0:02.49 kworker/u2:4
    1 root      20   0    2324    600    536 S  0.0  0.4   0:01.18 init
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.01 kthreadd
    3 root      20   0       0      0      0 S  0.0  0.0   0:00.26 ksoftirqd/0
    4 root      20   0       0      0      0 S  0.0  0.0   0:00.16 kworker/0:0
    5 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/0:0H
    6 root      20   0       0      0      0 S  0.0  0.0   0:02.13 kworker/u2:0
    8 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcu_sched
    9 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcu_bh
   10 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 khelper
   11 root      20   0       0      0      0 S  0.0  0.0   0:00.02 kdevtmpfs
   12 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 netns
   13 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 perf

root@swi-mdm9x28-wp:~# cm info
Device: WP7605
IMEI: 353532100020465
IMEISV: 6
FSN: 2A917285070404
Firmware Version: SWI9X07Y_02.28.03.05 000000 jenkins 2019/07/08 11:04:16
Bootloader Version: SWI9X07Y_02.28.03.05 000000 jenkins 2019/07/08 11:04:16
MCU Version:
PRI Part Number (PN): 9908705
PRI Revision: 002.006
Carrier PRI Name: SIERRA
Carrier PRI Revision: 001.032_000
SKU: 1104208
Last Reset Cause: Power Down
Resets Count: Expected: 59 Unexpected: 0
root@swi-mdm9x28-wp:~#
root@swi-mdm9x28-wp:~# legato version
19.02.0_a2f3e0fd302634e5deaac258dcf9b3fd

tomalex · December 8, 2020, 5:19pm

Hey,

Thanks for the quick response, sorry for my belated answer!
The test application is writing around 200x times in one second with PushBoolean, PushNumeric etc functions to the DataHub. DataHub does not do any observations.
dhub line from top command
“1017 root 20 0 4996 1868 1116 R 19.6 1.2 1:04.46 hubd”
test application has around the same ~20%. CPU usage.

I think it must be something related to interface api. Is there a limit/recommendations on the number of API calls?

Thanks,
tom

jyijyi · December 9, 2020, 12:04am

have you tried to add some sleep()?
Is that all IPC communictation (e.g. helloIPC sample) will have such behavior ?

tomalex · December 10, 2020, 12:53pm

Hi, thanks for the feedback.
There is no sleep() or similar anywhere. And as the CPU consumption seems to be symmetrics ~20% on Datahub, ~20% on the client application I think it is something related to interface.

Do we have some performance details on the interface API calls?

I will check the helloIPC sample and try out different legato versions, but probably I can only do it next week.

Thanks,
tom

jyijyi · December 10, 2020, 1:01pm

OK, wait for your test result.

BTW, you can limit the CPU usage by the following:

tomalex · February 17, 2021, 11:34am

Hello,

We have performed some test cases with a dedicated application. It shows us that there is indeed a performance issue dedicated probably to the legato IPC API calls. We are using Legato 20.04 on WP76xx. I don’t think limiting the CPU power should be a good solution. Is this issue with the API is known? Maybe that is one of the reason the Legato RPC is being developed?Please find our test application here:dataHubTest.zip (382.2 KB) ?
One test result:

We would appreciate any help!

Best,
tom

jyijyi · February 17, 2021, 1:19pm

Do you see this problem in helloIPC sample application?
Or this is just happened in your application which called datahub?

tomalex · February 17, 2021, 3:20pm

Hi,

Here is the slightly modified HelloIPC sample which sends 10 calls per every 100 msec. Which is not that many.HelloIpcLoop.zip (31.7 KB)
We already have 1-2% CPU usage per applications:

It seems it is indeed an API performance problem, and not related to DataHub.

jyijyi · February 17, 2021, 3:50pm

What do you expect for the cpu usage when doing 10 calls every 100ms? (which means 100 communications every second)

If you think it is too high, why don’t you limit the CPU usage in adef file?

tomalex · February 17, 2021, 4:13pm

Are you suggesting this should be normal for the legato API call to cause this kind of CPU load? I would not consider 10 calls per 100 ms too many on a powerful processor like this. If it is a known and expected behaviour I presume we have to live with this or try to improve it somehow.

Thx for suggesting limiting the CPU usage but it won’t really solve the root cause and might add unneeded delays.

jyijyi · February 18, 2021, 12:09am

I don’t really know if this is normal as IPC is something about communications between application.

You can see below and try other methods such as tcp communications or domain socket and see if they have better performance: