I understand the value of in-depth security reports, but the 5th time they told me "WolfsBane is the Linux counterpart of Gelsevirine, while FireWood is connected to Project Wood." I was wondering when I'd get to the meat and potatoes.
The report mentions: "we conclude ... exploited an unknown web application vulnerability ... ."
The chain of events, post initial exploit, is all very well but what was the initial point of entry? The IoCs etc are welcome - thanks.
/lib/systemd/system/display-managerd.service
And a process called "kde".
Security companies attribute activity based on their observations. ESET- a Slovakian company- is no exception.
> The WolfsBane Hider rootkit hooks many basic standard C library functions such as open, stat, readdir, and access. While these hooked functions invoke the original ones, they filter out any results related to the WolfsBane malware.
I took this to mean some things like a simple “ls -a” might now leave out those suspicious results.
Where is the backdoor coming from? If there's a backdoor, something is backdoored. An unknown exploit installing a rootkit and using a modified file, like usbdev.ko, is not a backdoor.
Which pakage / OS ships with the backdoor?
Or doesn't the author of TFA know the definition of a backdoor? Or is it me? I mean, to me the XZ utils exploit attempt was a backdoor (for example). But I see nothing here indicating the exploit they're talking about is a backdoor.
It reads like they classify anything opening ports and trying to evade detection as "backdoors".
Am I going nuts?
> A backdoor is a typically covert method of bypassing normal authentication or encryption
Not all malware acts like a backdoor: some malware exfiltrates data, some seeks to destroy the system, some encrypts data to hold it hostage, some performs attacks on other systems using your CPU/IP/memory, etc. The malware they are describing here does act like a backdoor though, and doesn't seem to have other malicious behavior.
The article says they don’t know how the attacker gets access to install this back door in the first place.
It doesn't really matter, because it's orthogonal. Malware like this can be installed on a system through any exploit that provides sufficient access.
So there's two parts to defending against it: 1) finding and fixing any vulnerability that allows the installation of malware like this, and 2) since #1 is a never-ending task, knowing about this malware so you can look specifically for it and delete it when you find it.
How it gets onto the system in the first place is the interesting (and dangerous) part, that sadly gets skimmed over here.
- intended target is KDE and GNOME
- privilege escalation through LD_PRELOAD hooking from userland via open, stat, readdir access (of any other program that the user executes, see down below)
- persistence through display manager config for KDE
- persistence through desktop autostart files for GNOME
- fallback persistence through .bashrc, profile or profile.sh in /etc
- installs trojanized ssh client version
- installs a JSP webshell
- sideloads kernel module as libselinux.so and .ko module. Probably the rootkit helpers to access them from userland
Despite the snarky comments in here, this malware is actually quite sophisticated.
If you don't agree, I challenge you now to measure the time it takes for you to find all .so files on your system that are loaded right now, and have been modified since your package manager installed them.
My point being that there is no EDR on Linux that catches this (apart from ours that's WIP), because all existing tools are just checking for windows malware hashes (not even symbols) as they're intended for linux fileservers.
Apologies for the dumb question, but does this mean my PiHole running on a small Debian server that has no KDE or Gnome installed, is safe from this?
(Personal opinion: You should never run Tomcat, history of really unsecure development practices)
NixOS with impermanence isn't exactly a security solution, but it does have some nice properties as a base for a secure system. It's also far from the only or even most notable of the immutable Linux systems, with abroot and RPM-ostree seeing more usage. It's probable there will still be some use in endpoint security but I suspect it will be greatly diminished if immutable image-based deployments with a secure boot chain is perfected on the Linux desktop system, for similar reasons to why it isn't really as important on ChromeOS either.
I also understand that this is not a silver bullet, as it's still possible on desktop Linux to persist malware through the bashrc/etc. like this does. Personally I blank out my home directory shell profile files and keep them root-owned to try to prevent them from being used for anything. It seems like more work is needed in the Linux ecosystem to figure out what to do about attack vectors like these, though I think a proactive approach would be preferred. (Scanning for infected files is always going to be useful, but obviously it's better if you stop it from happening in the first place.) In some ways the Linux desktop has gotten a bit more secure but many things are still not sandboxed sufficiently, which is a shame considering all of the great sandboxing technology available on Linux.
Anything else is probably going to be immeasurable security theatre.
I already replied somewhat in a sibling comment [1] in regards to the conceptual problem the article/malware focusses on.
In addition to that I think that it's a bad design choice relying so much on coreutils, binutils, and glibc behavior. A lot of those tools were written in a time when you trusted the system 100%, when there wasn't even an internet or downloadable programs yet.
In reality, it's unfeasible to have a user and group for each ELF binary/program that runs on your machine. Just managing file system access to all shared objects that each binary requires is a nightmare. AppArmor and other tools often go the "as good as possible" route here, but a lack of thorough profiling of binaries is usually the culprit why they can be exploited even in non-standard infrastructure systems.
The only way forward in my opinion is behavioral and network profiling (and the correlation between those) via eBPF/XDP. This way you can at least get the data to test against in those scenarios, whereas with AppArmor it's forensics that happened too late - long after you've been pwned, and you realized after hours of debugging that a rule was XOR ing another one with an unexpected side effect.
All these things that we are talking about are maintenance burden for the maintainers of the upstream distro, that in part have hundreds of "soft forks" of upstream software laying around that changes their behavior to reduce those attack surfaces. Even just little things like removing the SUID flag from packaged binaries becomes a huge burden to the maintainers, which in my opinion, should not even exist as a problem anymore.
Access to important (IAM related) things, like a keepassXC database file or the ssh keys somewhere in /home, should not be accessible to outside processes other than those that require it. The reality is though, discord can get pwned via HTML XSS messages and can just read it, and you wouldn't even know about it.
We need a better sandboxing system that by default denies access to anything on the filesystem, and requires mandatory rules/profiles what is able to be accessed. And that, hopefully, without more maintenance burden. This also implies that we have to get rid of all that $PATH related bullshit, and /usr/local and .local shenanigans have to be completely removed to get to a state where we can also call it a rootless base system.
POSIX as a reference to build your distro against isn't enough. Distros like Alpine focus on memory offsets and making exploitation of C-based software harder, but are useless once you realize everything is running as root anyways because they stop at "if you get pwned, you gotta reboot the container", so they're useless as a Desktop environment.
The issue I have with all these things is that there's this survivor's bias of Linux Desktop users that are not aware of how unsecure their system actually is. That's part of the reason why the late malware campaign trends of large APTs (APT3/APT28/APT29 etc) were so successful in targeting developer environments. They simply don't know that an "lsof" can be anything, and not the program they wanted to execute in the first place.
Anyway, this is how you check which open files match ".so" and see if they are modified since installation:
lsof | grep -o "/[^ ]*\.so[^ ]*" | while read path ; do
pkg=$(rpm -qf "$path" 2>/dev/null)
if [ $? != 0 ] ; then
echo "$path does not belong to a package"
else
rpm -V $pkg | grep -F "$path"
fi
doneHow does that work with the various namespaces? From root namespace you should see everything. But in a mount namespace you could bind mount under a different name. How would that confuse things? With a SELinux module even root cannot do everything. If /proc is mounted privately does it change anything?
Not sure, just starting to think. Linux has become incredibly complex since the old days...
Edit: orc routinely loads executable sections not belonging to any package.
sudo lsof | grep -o '/[^ ]*\.so[^ ]*' | awk '!seen[$0]++' | while read -r path; do
if p=$(dpkg -S "$path");then
cut -f1 -d: <<<"$p"
fi
done | awk '!seen[$0]++' | xargs -n1 debsums -s
No, yes(-ish)
Filenames are just a convention and not necessarily enforced.
lsof will really list every file the kernel thinks has a handle held by an active process, but depending on your threat model I think you could get around this. For example you could copy the malicious code into memory and close the file, or use your preload to modify the behaviour of lsof itself (or debsums).
Anyway debsums is a great tool. I'd have used a command similar to yours, though maybe run debsums first, use `file` to filter dynamic libraries and then check which of those have been recently accessed from disk.
Other than that it seems to work - i.e. no alarming output. :-)
Do not trust key material, sensitive data or remote logins that the backdoored system have had control over. Repeat the same operation for them.
To check for backdoors, again boot from rescue media and do a full integrity check. Do not limit the check to open files.
Your solution failed because you installed the rootkit that was aliased via the lsof command in the .bashrc.
Additionally, lsof like so many other tools rely on procfs, which allows processes to rewrite their own process names (comm) and arguments (cmdline).
Even if the malware of the article would run only in userspace (as non-root and "only a wheel user"), you certainly would have executed it.
My point being that you also forgot to check any processes against environment variables like LD_PRELOAD that the malware uses before executing any command (meaning even syntax programs like "if" as a program can be hijacked).
Again, this is a conceptual problem because there is a lot of programs in $PATH that can be executed by the same user, meaning only a kernel hook or ebpf module can audit/grant access to these kind of things to prevent that.
There is no trusted execution in Linux because of so many things down the line. Glibc, the $PATH mess, aliases, .local overrides etc.
Checking environment variables wasn't part of the challenge. The challenge was likely not intended to check for rootkits, because there are a thousands of other ways to place a rootkit apart from already loaded libraries. (Why check only open files? Closed files can also contain unwanted things.)
If the purpose is to check system integrity, just check all packages. That is much easier and faster.
If there is even the slightest possibility that the system is already compromised, do it from rescue media.
Couldn't we have a nice overview of what kind of signed modules are valid in their integrity and authenticity based on cryptography?
(Also I wanted to point out that LD_PRELOAD was specifically mentioned in my comment, but it doesn't really matter, it's the lack of integrity checks across the /usr folders that are part of the problem. Glibc, $PATH, sideloaded .so files, kernel hooks...it's such a vast problem space of insecure development practices that by now we need a better OS architecture because all (old) tools down the stack rely on 100% trustable programs being installed, which after the invention of the internet is not a reality anymore.)
Perhaps more interestingly than how long it takes, some of the files mapped to memory are already deleted. They should have been checked at the time of loading, not hours or days later, when it's not longer possible.
As long as processes can rewrite their own cmdline and process names, you have a conceptual problem that you can only solve with kernel hooks (or eBPF modules).
The persistence techniques in the article were easy to follow, but all that alias mess, path mess, and glibc dependent mess makes everything that you execute untrustable.
The cli commands that were posted in the sibling comments all rely on procfs and the faked names :) so they won't actually detect it if a process rewrote its cmdline or has an in memory .so file that was changed and loaded from somewhere else (e.g. via LD_PRELOAD).
LD_PRELOAD is quite easy to detect, though nobody seems to be aware of its effects. And that is a 10 years known vulnerability and part of every standard audit by now. None of the posted answers even check for the environment files in procfs.
We're not talking about a bug in glibc here, because it is intended and documented behavior. If it was a bug, it would be much much worse.
edit: I wanted to add that the POSIX and Linux way of doing things would require a specific user for each program in order to be successful. But this is a prime example of what can go wrong when a user (and its groups) is used for multiple things. Any process that is running as the same (non-root) user can modify those procfs files. And I think that's a HUGE problem.
/var/*/*maps
?The article doesn't mention how the rootkit ended up on the machines in question, it seems to indicate a vulnerable web application. I wish I knew which one.
They are certainly not trying to hide, and it has nothing to do with the initial exploit surface. You are mixing up things because you seem to be not aware how multi stage exploits work.
The article also mentioned that tomcat was targeted, but it didn't mention whether it was a zero day or a known vulnerability (like log4j vulnerabilities, for example).
- The initial access stage of that malware was a Tomcat exploit
- The privilege escalation stage was done via both userland + kernelmod, whereas userland's method was using glibc, hijacking any open() call of any process that is executed later. If you enter your sudo password any time for anything after that in any bash shell, it's escalated successfully and can install the kernel mod (that's what the .bashrc and profile entries were for).
- The persistence stage was done with a kernel mod that can now pretty much do whatever it wants.
Edit: After looking a little bit further, the initial access exploit was very likely CVE-2024-52316 [1], which is a Tomcat bug specific for that Jakarta Authentication system, given the described georegional malware campaign targets.
Sorry but not all of us have formal education in this field, I'm just trying to understand if you're saying that KDE and Gnome systems are vulnerable, or Tomcat web servers.
> Although we lack concrete evidence regarding the initial access vector, the presence of multiple webshells (as shown in Table 1 and described in the Webshells section) and the tactics, techniques, and procedures (TTPs) used by the Gelsemium APT group in recent years, we conclude with medium confidence that the attackers exploited an unknown web application vulnerability to gain server access.
as a target demographic, the folks using linux on the desktop are almost certainly more technical than most, and are likely developers, engineers, admins, or otherwise STEM types. probably high overlap with domains like SRE and security.
also probably, as a whole, better paid, and more likely to have cryptocurrency.
In this case for example the attackers tried to hide their files by disguising them as other known file paths on the system.
If you use a tripwire setup you will get an alert when a file appears that is not supposed to be there. Of course this requires a more hands-on approach where you create excludes for all your applications.
There is also snort which is a more libre project, but it's more of a full featured IDS that try to sell subscriptions for patterns. Think of them sort of like virus definitions but for rootkits and intrusions.
You can technically setup Snort as a tripwire.
A tripwire is very simple, some people have made them from scratch using Cronjobs and shell scripts. They simply maintain a database of all your files and their checksums, and alert you when a checksum changes.
But security is more than just an IDS. I would recommend SElinux+IDS+remote logging+MFA+granular user security and more!