Lessons learnt from using Snyk Vulnerability Management for containers
If you think that Sonar Qube in your CICD pipeline is enough then you are wrong. Usually you’ve got unmaintained Linux distribution (userspace tools) in your aging docker container running in production. Java inside container is also static. You need to be aware of vulnerabilities, risks and mitigations.
Let’s consider Spring Boot 2.0.x microservice created in 2018. Fat jar running on Java directly from init/entrypoint script. Was the Java base image scanned for vulnerabilities by the creators? Were they using latest Ubuntu image bootstrapped with apt-get update? These are very important aspects of your IT security. You need to have a governance process for docker base images.
What do we see here? 15 high severity bugs, 274 medium and 342 low. Definitely too much. We need to check for false positives, estimate the real risk and provide mitigations. It would be good if we could eliminate some categories by a system approach.
There is a security bug in apt package. Man in the middle attack is possible during apt-get install/update execution. We shouldn’t have apt-get in a production container. The content should be immutable. We often leave a lot of tools just in case we would need to troubleshoot a production problem, however this is a security anti-pattern. MITM forces us to think about zero trust network. Are you sure that Kubernetes overlay network is encrypted in Azure data center? Can a hacker override your http_proxy environment variable and redirect a traffic?
For every detected vulnerability Snyk provides a score — how serious it is. Attack vector in case of apt is network, complexity is high however none privileges are required a no user interaction is needed. This can be an automated attack, so this is serious.
Snyk gives you a hint which item should be addressed earlier. If severity is high and fix is available you should not wait. Remember: you can always have a short term and long term solutions.
Curl is often used by an internal docker alive check. If you use Kubernetes you can think about external HTTP(s) check. However here we see TFTP protocol, which we don’t use. So for us this is a false positive.
Why do we have a filesystem tool for ext2/ext3/ext4 in a Java container? It’s totally not needed.
Linus Torvalds doesn’t like Systemd and he’s probably right. Systemd is for daemon management in a real Linux. I think that in a specialized Java container we don’t need it.
Exploit maturity: no known one, and severity is low. It would be good to know how this vulnerability can be detected. We can be prepared for 0day exploit or update curl or remove it.
There is a mature exploit and this is related to bash and UIDs, so together with containers effective severity may not be so low. We may consider mitigating this earlier than later.
Bug in Simple Authentication and Security Layer of OpenLDAP sounds bad. If we do use LDAP inside container everything related to authorization should be treated seriously.
There is no end of speculative execution bugs in Intel silicon. AMD and ARM were also affected in some cases. Snyk detects GCC without mitigations, however Snyk is slightly wrong here — every binary compiled with this GCC version is vulnerable to data stealing attack, or every library using standard C++ compiled with affected GCC/G++. The scope is very wide. But Snyk doesn’t recognize platforms. We are on x86–64, not ARMv8. We are safe.
An attacker can supply malformed host name and cause denial of service attack due to running out of free socket/file descriptors. Bug is in C library and inside all programs using it, also Java.
Copying memory regions by C library on Intel with AVX-512 instructions (basically every cloud machine) might create a buffer overflow — scary…
If you use PGP, not via Bouncy Castle in Java, but via local command line tools, you should update GnuPG above 2.2.11.
If the bug is in gnutls it may affect your Java HTTPS connections. Man in the middle would be able to recover plain text data. Remember: don’t trust the network, use TLS, patched TLS.
Cross-VM attack means that if your cloud provider’s hypervisor doesn’t mitigate attacks against Intel CPU/Hyperthreading then side channel attack to recover plain text data is possible. From other container running on the same physical core. This is very interesting: if your order only 1 Azure VM with 1 vCPU it must occupy full physical core; if you order 2 Azure VMs, each with 1 vCPU — they should be scheduled on the same physical core (2 vCPUs thanks to HT); if you pay for Azure VM with 2 vCPUs workload is bound to one physical core. Cloud schedulers (in hypervisor, Kubernetes) have got a lot of work to do. I think that their engineers-programmers hate Intel.
Secure computing library to filter and block some system calls is used by Red Hat OpenShift for securing containers. Inside container these bugs are low priority, on the host they are serious.
HTTP/2 and HTTP/3, as a something relatively new, are expected to have bugs. Affected here is really curl as a client, so for me this is a low severity.
If I have a WebUI using input provided by remote users and my Java uses GNU libC regular expressions, I would be worried. Java however uses own regexp engine. So low severity.
There is a bug allowing to bypass Address Space Layout Randomization. Nowadays all OSes support ASLR and exploits don’t expect to have functions at fixed addresses. Low severity.
Bug allowing to fool PGP signature verification is serious. It shows that SHA-1 is nowadays considered too weak due to collisions. Just use at least SHA-384, with SHA-512 preferred.
An attacker who is able to run process on the same physical core as the victim process, could use a situation to extract plain text or in some cases downgrade any TLS connection to a vulnerable server. To mitigate this correctly we need patched CPU microcode, patched hypervisor (Hyper-V, VMware) and probably patched userspace. Pandora’s box is still open and probably will be. If you disable HT on Intel Xeon from 128 vCPUs suddenly you get only 64. Cloud provider can increase CPU overcommit (read: decrease performance of your workloads) or buy additional hardware with huge discounts from Intel… or patch CPU scheduling.
Why do we have a perl in a Java container? We really need some small Java docker base image.
Do we need a mount command in a container? Unlikely. To effectively get: mount -o remount,ro /, we can set correct filesystem permissions instead.
There is Red Hat Linux which has got SE Linux and some additional security features enabled in GCC (FORTIFY_SOURCE) and system binaries. These may effectively stop some vulnerabilities. Think about Red Hat Universal Base Image.
Sudo inside a container is a bad thing. Please do not include it and do not run container processes as a root :)
When you want to mitigate risks by removing unneeded software please use check dependency tree.
What’s the most important thing here: we must scan containers every time vulnerability database is updated, at least every day. Static code analysis, Veracode scan, Black Duck scan inside CI/CD pipelines are needed during creation of the container, but vulnerability scanning is also needed during the period when the container is alive in production environment. 2 separate aspects, but the last one is often overlooked/skipped.
Snyk is a good tool, but doesn’t have a full coverage. Java distributions, Java frameworks are not recognized. These are closer to the attacker than unused userspace programs inside container. At least when we use containers to host microservices.
