How to analyze cloud memory dump from RHEL8/Ubuntu 20+ using Volatility 3
First of all we need a modern and maintained tool to dump memory and do it over the network. Recommended one is from Microsoft: avml /dev/stdout | aws s3 cp — s3://dfir/memory. Microsoft must have it because they ported Hyper-V to Linux and run container/Kubernetes on Linux in a shared model — many customers on one node. AWS uses in some cases Firecracker and can enforce container isolation via KVM. It is more expensive but also more secure.
We are going to analyze memory dumps using Volatility 3, not 2. Why? Version 2 requires building a special kernel module with kernel sources and then extracting kernel symbols. This requires access to the original operating system. It is a big overhead. Version 3 allows to use definitions of kernel symbols extracted from vmlinux compiled in debug mode and System.map. These can be easily downloaded from the Internet and unpacked from packages (rpm, deb). The problem is that so called Linux banner taken from a debug kernel is different than Linux banner in RAM (non-debug kernel). Volatility 3 uses direct string match and we must fix symbol definition.
A description how to prepare the environment is below:
yum install python3-pip python3-devel -y
pip3 install leechcorepyc PyCrypto yara-python
git clone https://github.com/volatilityfoundation/volatility3
cd volatility3
python3 setup.py build
python3 setup.py build install
cd ..
yum module -y install go-toolset
git clone https://github.com/volatilityfoundation/dwarf2json
cd dwarf2json/
go build
cd ..
cd volatility3
python3 vol.py -vvvv -f /home/dfir/memory_ip-192–168–100–14_20210922–1423.lime bannersVolatility 3 Framework 1.2.1…
[here you should see at least one entry similar to uname -a output]Offset Banner0xf550649 Linux version 4.18.0–305.17.1.el8_4.x86_64 (mockbuild@x86-vm-09.build.eng.bos.redhat.com) (gcc version 8.4.1 20200928 (Red Hat 8.4.1–1) (GCC)) #1 SMP Mon Aug 30 07:26:31 EDT 20210x18d3248a Linux version 4.18.0–305.17.1.el8_4.x86_64 (mockbuild@x86-vm-09.build.eng.bos.redhat.com) (gcc version 8.4.1 20200928 (Red Hat 8.4.1–1) (GCC)) #1 SMP Mon Aug 30 07:26:31 EDT 20210x18d3256a Linux version 4.18.0–305.17.1.el8_4.x86_64 (mockbuild@x86-vm-09.build.eng.bos.redhat.com) (gcc version 8.4.1 20200928 (Red Hat 8.4.1–1) (GCC)) #1 SMP Mon Aug 30 07:26:31 EDT 20210x1ac00100 Linux version 4.18.0–305.17.1.el8_4.x86_64 (mockbuild@x86-vm-09.build.eng.bos.redhat.com) (gcc version 8.4.1 20200928 (Red Hat 8.4.1–1) (GCC)) #1 SMP Mon Aug 30 07:26:31 EDT 20210x1bcb9714 Linux version 4.18.0–305.17.1.el8_4.x86_64 (mockbuild@x86-vm-09.build.eng.bos.redhat.com) (gcc version 8.4.1 20200928 (Red Hat 8.4.1–1) (GCC)) #1 SMP Mon Aug 30 07:26:31 EDT 20210x22f589a8 Linux version 4.18.0–305.17.1.el8_4.x86_64 (mockbuild@x86-vm-09.build.eng.bos.redhat.com) (gcc version 8.4.1 20200928 (Red Hat 8.4.1–1) (GCC)) #1 SMP Mon Aug 30 07:26:31 EDT 2021cd ../dwarf2json## Now we need System.map
## Let's assume it's in /boot/System.map-4.18.0–80.4.2.el8_0.x86_64##wget http://debuginfo.centos.org/8/x86_64/Packages/kernel-debug-debuginfo-4.18.0-305.17.1.el8_4.x86_64.rpmwget https://oss.oracle.com/ol8/debuginfo/kernel-debug-debuginfo-4.18.0-305.17.1.el8_4.x86_64.rpmrpm -i kernel-debug-*.rpm — nodeps./dwarf2json linux — elf /usr/lib/debug/lib/modules/4.18.0–305.17.1.el8_4.x86_64+debug/vmlinux — system-map System.map-4.18.0–305.17.1.el8_4.x86_64 > rhel8_4.18.0–305.17.1.el8_4.x86_64.jsonmkdir ../volatility3/volatility3/symbols/linuxmv rhel8_*.json ../volatility3/volatility3/symbols/linuxcd ../volatility3python3 vol.py -vvvvvv -f /home/dfir/memory*.lime isfinfo…URI Valid Number of base_types Number of types Number of symbols Number of enums Windows info Linux banner Mac bannerfile:///root/dfir/volatility3/volatility3/symbols/linux/rhel8_4.18.0–305.17.1.el8_4.x86_64.json Unknown 18 9841 157524 1582 — Linux version 4.18.0–305.17.1.el8_4.x86_64+debug (mockbuild@kbuilder.bsys.centos.org) (gcc version 8.4.1 20200928 (Red Hat 8.4.1–1) (GCC)) #1 SMP Wed Sep 8 13:46:08 UTC 2021� -file:///root/dfir/volatility3/volatility3/framework/symbols/generic/qemu.json Unknown 7 2 0 0 — — -file:///root/dfir/volatility3/volatility3/framework/symbols/linux/bash32.json Unknown 2 1## Now we need to adjust Linux bannercat volatility3/symbols/linux/rhel8_4.18.0–305.17.1.el8_4.x86_64.json | grep -A 10 “linux_banner” | grep constant | cut -d\” -f4 | base64 -dLinux version 4.18.0–305.17.1.el8_4.x86_64+debug (mockbuild@kbuilder.bsys.centos.org) (gcc version 8.4.1 20200928 (Red Hat 8.4.1–1) (GCC)) #1 SMP Wed Sep 8 13:46:08 UTC 2021BANNER2=`echo “Linux version 4.18.0–305.17.1.el8_4.x86_64 (mockbuild@x86-vm-09.build.eng.bos.redhat.com) (gcc version 8.4.1 20200928 (Red Hat 8.4.1–1) (GCC)) #1 SMP Mon Aug 30 07:26:31 EDT 2021” | base64 | tr -d ‘\n’`BANNER1=`cat volatility3/symbols/linux/rhel8_4.18.0–305.17.1.el8_4.x86_64.json | grep -A 10 “linux_banner” | grep constant | cut -d\” -f4`cat volatility3/symbols/linux/rhel8_4.18.0–305.17.1.el8_4.x86_64.json | sed -e s/$BANNER1/$BANNER2/ > volatility3/symbols/linux/manual_rhel8_4.18.0–305.17.1.el8_4.x86_64.jsonrm volatility3/symbols/linux/rhel*.*## The same replacement can be done for Ubuntu
