I’d be surprised if this is the cause, though, because both supervisor and sdir are running as root (with CAP_MAC_ADMIN and CAP_MAC_OVERRIDE privileges). There’s nothing in /legato/smack/onlycap right?
Correct. /legato/smack/onlycap is empty.
I had some trouble building for the virt target. I grabbed the latest Legato from GitHub using repo, but following the directions you provided, I get the following error:
FAILED: /opt/swi/y22/i586-poky-linux-gcc --sysroot=/opt/swi/y22/sysroots/i586-poky-linux -MMD -MF /home/jchitty/legato-repo/legato/build/virt/framework/obj/dir.c.o.d -c /home/jchitty/legato-repo/legato/framework/c/src/dir.c -o /home/jchitty/legato-repo/legato/build/virt/framework/obj/dir.c.o -Wall -fPIC -Werror -O2 -DLE_COMPONENT_NAME=framework -I/home/jchitty/legato-repo/legato/framework/c/inc -DDISABLE_SMACK=0 -DLE_SVCDIR_SERVER_SOCKET_NAME="\"/tmp/legato/serviceDirectoryServer\"" -DLE_SVCDIR_CLIENT_SOCKET_NAME="\"/tmp/legato/serviceDirectoryClient\"" -DLEGATO_EMBEDDED
/bin/sh: 1: /opt/swi/y22/i586-poky-linux-gcc: not found
I tried setting
export VIRT_TOOLCHAIN_DIR=/opt/swi/y22/sysroots/x86_64-pokysdk-linux/usr/bin/i586-poky-linux
and this got further, but failed when mksys ran:
mksys -t virt -w build/virt/system -o build/virt default.sdef \
--cflags=-O2
** WARNING: Invalid sysroot returned from compiler '/opt/swi/y22/sysroots/x86_64-pokysdk-linux/usr/bin/i586-poky-linux/i586-poky-linux-gcc' (returned '/not/exist').
[35/658] Running external build step
FAILED: CC=/opt/swi/y22/sysroots/x86_64-pokysdk-linux/usr/bin/i586-poky-linux/i586-poky-linux-gcc CXX=/opt/swi/y22/sysroots/x86_64-pokysdk-linux/usr/bin/i586-poky-linux/i586-poky-linux-g++ CFLAGS="-O2 -I build/virt/system -I/home/jchitty/legato-repo/legato/interfaces -I/home/jchitty/legato-repo/legato/framework/c/inc -I. -Iinterfaces/modemServices -Iinterfaces/positioning" CXXFLAGS="-I build/virt/system -I/home/jchitty/legato-repo/legato/interfaces -I/home/jchitty/legato-repo/legato/framework/c/inc -I. -Iinterfaces/modemServices -Iinterfaces/positioning" LDFLAGS="" cd build/virt/system/component/8f7bd0cfe27f5bae6c1fe07c62b7db39; cd /home/jchitty/legato-repo/legato/3rdParty/Lwm2mCore/tinydtls && autoreconf -i
/bin/sh: 1: autoreconf: not found
[35/658] Compiling C source
ninja: build stopped: subcommand failed.
Makefile:400: recipe for target 'system_virt' failed
make: *** [system_virt] Error 1
It turned out I was missing autoconf and automake from my system (those are not listed in the Legato install/build instructions on legato.io).
After installing those, it built. Now to try installing it…
Right. Can’t do update or app install because the Legato framework isn’t running. 
It definitely seems to be SMACK-related. If I start the serviceDirectory and logCtrlDaemon from the command-line (in that order), then ‘sdir list’ runs fine. I can then start the configTree manually and ‘sdir load’ runs fine. But, if I let the supervisor start the daemons, then ‘sdir list’ can’t connect to the logCtrlDaemon.
I looked around for bugs and patches related to sending file descriptors over unix domain sockets and found this one. So I downloaded the kernel sources and had a look. The patch is in the 4.8.12 kernel, so that’s not it. Will keep looking tomorrow.
When do you launch the processes this way, what are their smack labels?
_ (floor)
In this case, they all have the same label.
I’ve confirmed that sockets are indeed getting the wrong label. They all seem to be getting ‘_’, regardless of the label on the process that created them. This appears to be a kernel bug. We can work-around the problem to get the framework and unsandboxed apps working, but sandboxed apps won’t work until the kernel bug is fixed.
A workaround to get the framework to start up:
echo framework _ rw > /legato/smack/load2
echo _ framework rw > /legato/smack/load2
This is NOT a good final solution, though, because it violates the principal of least privilege.
For now, that will let us move forward with our planning and prototyping.
Thank you Sierra for the time and attention to this matter. I know this isn’t a simple problem to track down.
OK, so I think I’ve found the problem. The documentation for this LSM-related stuff is really pretty bad. In 2005, a new Linux-specific socket option flag SO_PASSSEC and a new ancillary data message SCM_SECURITY were added (it looks like part of the SELINUX development). This was not documented and remains undocumented and even (in the case of SCM_SECURITY) unsupported in the libc headers on many modern systems, including this 4.8.12-based Yocto release. But, if this socket option is not set, then passing a file descriptor over that socket will not include the “security context”, which in our case is the SMACK label. The result is that the receiving end doesn’t have the SMACK label that should be attached to the file descriptor it receives, so it defaults to “_”, which the receiving process is not allowed to write to, so the SMACK security check rejects the received file descriptor.
In older kernels, the SMACK security checks for file descriptor passing on unix domain sockets were either not implemented or broken, so our stuff just worked without setting the SO_PASSSEC flag on the socket. But, at some point quite recently (before kernel 4.8) it was fixed. (Actually, it was broken first, then fixed in December 2015 by commit 79be093500791cc25cc31bcaec5a4db62e21497b.)
Anyway, we now need to make a few changes to to get things working on these newer kernels:
- Add SCM_SECURITY to the standard library headers;
- in unixSocket.c, in unixSocket_CreateSeqPacketUnnamed(), set the SO_PASSSEC socket option flag;
- In unixSocket.c, in ExtractAncillaryData(), handle the SCM_SECURITY message (just throw it away);
- Make the Supervisor add a couple of new SMACK rules:
- “framework admin w”
- “admin framework w”
- Set the extended attribute “security.SMACK64EXEC” to “framework” on all command-line tools bundled with the framework.
The good news is that we don’t have to patch the kernel to get this working. Although, we should probably submit a patch to fix the documentation.
Fantastic news! And, an impressive rundown of under-documented features.
I’ll be happy to test any new commits on my end, or patch the code together myself for testing.
Using this toolchain that cholmes are using I was able to compile legato as virt and generate a squashfs file.
However when trying to execute the start file I came across the issue below:
sudo: unable to execute ./startSystem: No such file or directory
Have you faced it before?
The “No such file or directory” can appear if it’s not built for the right architecture.
Like if you build for x86 and you execute it on arm it would return that error as well (it’s a bit misleading …).
So an advise would to make sure that it builds for the right architecture, with the right toolchain.
Note that if you’re building for virt, by default there is that VIRT_TARGET_ARCH env variable that controls the target architecture. It takes either arm or x86, and defaults to arm.
Even though I’m using the configuration below, I’m still receiving that error.
export VIRT_TOOLCHAIN_DIR=/opt/swi/y22/sysroots/x86_64-pokysdk-linux/usr/bin/i586-poky-linux
export VIRT_TARGET_ARCH=x86
export LEGATO_SYSROOT=/opt/swi/y22/sysroots/i586-poky-linux
That is odd, which version of Legato are you building with? latest?
I have this:
file build/virt/staging/system/bin/startSystem
build/virt/staging/system/bin/startSystem: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=ffa85db5f395fd6ba5664f886feed0b51f7f6be8, not stripped
Another thing that is needed:
In framework/c/src/start/start.c, in the function MakeDir(), change the line
le_result_t result = le_dir_Make(dirPath, DEFAULT_PERMS);
to
le_result_t result = dir_MakeSmack(dirPath, DEFAULT_PERMS, "_");
This ensures that directories created by the start-up program get the correct SMACK label.