Use Volatility to find information about applications, network connections, kernel modules, files, and more.
The operating system and applications of a computer use main memory (RAM) to perform various tasks. This volatile memory contains a wealth of information about running applications, network connections, kernel modules, open files, and almost everything else, but this information is cleared every time the computer is restarted.
Memory forensics is a way to find and extract this valuable information from memory. Volatility is an open-source tool that uses plugins to handle this type of information. However, there is a catch: before you can work with this information, you must dump the physical memory to a file, and Volatility does not have this capability.
Therefore, this article is divided into two parts:
- The first part deals with acquiring physical memory and dumping it to a file.
- The second part uses Volatility to read and process this information from the memory dump.
I used the following test system in this tutorial, but it can work on any Linux distribution:
$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.3 (Ootpa)
$
$ uname -r
4.18.0-240.el8.x86_64
$
Note: Part 1 involves compiling and loading a kernel module. Don't worry: it's not as difficult as it sounds.
Some guidelines:
Follow the steps below.
Do not attempt any of these steps on a production system or your main computer.
Always use a tested virtual machine (VM) to try until you are familiar with using these tools and understand how they work.
Install Required Packages#
Install the necessary tools before starting. If you frequently use a Debian-based distribution, you can use the apt-get command. Most of these packages provide the necessary kernel information and tools to compile the code:
$ yum install kernel-headers kernel-devel gcc elfutils-libelf-devel make git libdwarf-tools python2-devel.x86_64-y
Part 1: Use LiME to Acquire Memory and Dump It to a File#
Before you start analyzing memory, you need a memory dump to work with. In actual forensic activities, this might come from a compromised or hacked system. This information is usually collected and stored to analyze how the intrusion occurred and its impact. Since you may not have an available memory dump, you can acquire a memory dump from your test VM and use it for memory forensics.
Linux Memory Extractor (LiME) is a commonly used tool for acquiring memory on Linux systems. Use the following command to obtain LiME:
$ git clone https://github.com/504ensicsLabs/LiME.git
$
$ cd LiME/src/
$
$ ls
deflate.c disk.c hash.c lime.h main.c Makefile Makefile.sample tcp.c
$
Build the LiME Kernel Module#
Run the make command in the src folder. This will create a kernel module with a .ko extension. Ideally, at the end of the make process, the lime.ko file will be renamed to lime-.ko.
$ make
make -C /lib/modules/4.18.0-240.el8.x86_64/build M="/root/LiME/src" modules
make[1]: Entering directory '/usr/src/kernels/4.18.0-240.el8.x86_64'
<< omitted >>
make[1]: Leaving directory '/usr/src/kernels/4.18.0-240.el8.x86_64'
strip --strip-unneeded lime.ko
mv lime.ko lime-4.18.0-240.el8.x86_64.ko
$
$
$ ls -l lime-4.18.0-240.el8.x86_64.ko
-rw-r--r--. 1 root root 25696 Apr 17 14:45 lime-4.18.0-240.el8.x86_64.ko
$
$ file lime-4.18.0-240.el8.x86_64.ko
lime-4.18.0-240.el8.x86_64.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=1d0b5cf932389000d960a7e6b57c428b8e46c9cf, not stripped
$
Load the LiME Kernel Module#
Now it's time to load the kernel module to acquire the system memory. The insmod command will help load the kernel module; once the module is loaded, it will read the main memory (RAM) on your system and dump the contents of memory to a file in the path directory provided in the command line. Another important parameter is format; keep the format as lime, as shown below. After inserting the kernel module, use the lsmod command to verify that it has indeed been loaded.
$ lsmod | grep lime
$
$ insmod ./lime-4.18.0-240.el8.x86_64.ko "path=../RHEL8.3_64bit.mem format=lime"
$
$ lsmod | grep lime
lime 16384 0
$
You should see that the file specified by the path command has been created, and the file size matches the size of your system's physical memory (RAM) (not surprisingly). Once you have the memory dump, you can use the rmmod command to remove the kernel module:
$
$ ls -l ~/LiME/RHEL8.3_64bit.mem
-r--r--r--. 1 root root 4294544480 Apr 17 14:47 /root/LiME/RHEL8.3_64bit.mem
$
$ du -sh ~/LiME/RHEL8.3_64bit.mem
4.0G /root/LiME/RHEL8.3_64bit.mem
$
$ free -m
total used free shared buff/cache available
Mem: 3736 220 366 8 3149 3259
Swap: 4059 8 4051
$
$ rmmod lime
$
$ lsmod | grep lime
$
What is in the Memory Dump?#
This memory dump file is just raw data, as you can see with the file command. You cannot manually understand it; yes, there are some ASCII characters in there, but you cannot open this file with an editor and read it. The output of hexdump shows that the initial few bytes are EmiL; this is because your requested format in the command line above is lime:
$ file ~/LiME/RHEL8.3_64bit.mem
/root/LiME/RHEL8.3_64bit.mem: data
$
$ hexdump -C ~/LiME/RHEL8.3_64bit.mem | head
00000000 45 4d 69 4c 01 00 00 00 00 10 00 00 00 00 00 00 |EMiL............|
00000010 ff fb 09 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 b8 fe 4c cd 21 44 00 32 20 00 00 2a 2a 2a 2a 2a |..L.!D.2 ..*****|
00000030 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a |****************|
00000040 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 20 00 20 |************* . |
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000080 00 00 00 00 00 00 00 00 00 00 00 00 70 78 65 6c |............pxel|
00000090 69 6e 75 78 2e 30 00 00 00 00 00 00 00 00 00 00 |inux.0..........|
000000a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
$
Part 2: Obtain Volatility and Use It to Analyze Your Memory Dump#
Now that you have a sample memory dump to analyze, use the command below to obtain the Volatility software. Volatility has been rewritten in Python 3, but this tutorial uses the original Volatility package written in Python 2. If you want to experiment with Volatility 3, you can download it from the appropriate Git repository and use Python 3 instead of Python 2 in the commands below:
$ git clone https://github.com/volatilityfoundation/volatility.git
$
$ cd volatility/
$
$ ls
AUTHORS.txt contrib LEGAL.txt Makefile PKG-INFO pyinstaller.spec resources tools vol.py
CHANGELOG.txt CREDITS.txt LICENSE.txt MANIFEST.in pyinstaller README.txt setup.py volatility
$
Volatility uses two Python libraries to implement certain functionalities, so use the following commands to install them. Otherwise, when you run the Volatility tool, you may see some import errors; you can ignore them unless the plugins you are running require these libraries; in that case, the tool will report errors:
$ pip2 install pycrypto
$ pip2 install distorm3
List Volatility's Linux Profiles#
The first Volatility command you will run lists the available Linux profiles, and the main entry point for running Volatility commands is the vol.py script. Call it using the Python 2 interpreter and provide the --info option. To narrow down the output, look for strings that start with Linux. As you can see, there are not many Linux profiles listed:
$ python2 vol.py --info | grep ^Linux
Volatility Foundation Volatility Framework 2.6.1
LinuxAMD64PagedMemory - Linux-specific AMD 64-bit address space.
$
Build Your Own Linux Profile#
Linux distributions are diverse and built for different architectures. This is why profiles are necessary — Volatility must know from which system and architecture the memory dump was obtained before extracting information. There are some Volatility commands that can find this information; however, this method can be time-consuming. To speed things up, you can use the following command to build a custom Linux profile:
Move to the tools/linux directory of the Volatility repository and run the make command:
$ cd tools/linux/
$
$ pwd
/root/volatility/tools/linux
$
$ ls
kcore Makefile Makefile.enterprise module.c
$
$ make
make -C //lib/modules/4.18.0-240.el8.x86_64/build CONFIG_DEBUG_INFO=y M="/root/volatility/tools/linux" modules
make[1]: Entering directory '/usr/src/kernels/4.18.0-240.el8.x86_64'
<< omitted >>
make[1]: Leaving directory '/usr/src/kernels/4.18.0-240.el8.x86_64'
$
You should see a new module.dwarf file. You also need the System.map file in the /boot directory because it contains all the symbols related to the currently running kernel:
$ ls
kcore Makefile Makefile.enterprise module.c module.dwarf
$
$ ls -l module.dwarf
-rw-r--r--. 1 root root 3987904 Apr 17 15:17 module.dwarf
$
$ ls -l /boot/System.map-4.18.0-240.el8.x86_64
-rw-------. 1 root root 4032815 Sep 23 2020 /boot/System.map-4.18.0-240.el8.x86_64
$
$
To create a custom profile, move back to the Volatility directory and run the command below. The first parameter provides a custom .zip file, which you name yourself. I often use the operating system and kernel version for naming. The next parameter is the module.dwarf file created earlier, and the last parameter is the System.map file in the /boot directory:
$
$ cd volatility/
$
$ zip volatility/plugins/overlays/linux/Redhat8.3_4.18.0-240.zip tools/linux/module.dwarf /boot/System.map-4.18.0-240.el8.x86_64
adding: tools/linux/module.dwarf (deflated 91%)
adding: boot/System.map-4.18.0-240.el8.x86_64 (deflated 79%)
$
Now the custom profile is ready, so check if the .zip file has been created in the location provided earlier. If you want to know if Volatility detects this custom profile, run the --info command again. Now, you should see the new profile listed in the output below:
$
$ ls -l volatility/plugins/overlays/linux/Redhat8.3_4.18.0-240.zip
-rw-r--r--. 1 root root 1190360 Apr 17 15:20 volatility/plugins/overlays/linux/Redhat8.3_4.18.0-240.zip
$
$
$ python2 vol.py --info | grep Redhat
Volatility Foundation Volatility Framework 2.6.1
LinuxRedhat8_3_4_18_0-240x64 - A Profile for Linux Redhat8.3_4.18.0-240 x64
$
$
Getting Started with Volatility#
Now you are ready to do some real memory forensics. Remember, Volatility is composed of custom plugins that you can use to obtain information from the memory dump. The general format of the command is:
python2 vol.py -f <memory-dump-file-taken-by-Lime> <plugin-name> --profile=<name-of-our-custom-profile>
With this information, run the linux_banner plugin to see if you can identify the correct distribution information from the memory dump:
$ python2 vol.py -f ~/LiME/RHEL8.3_64bit.mem linux_banner --profile=LinuxRedhat8_3_4_18_0-240x64
Volatility Foundation Volatility Framework 2.6.1
Linux version 4.18.0-240.el8.x86_64 ([[email protected]][4]) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)) #1 SMP Wed Sep 23 05:13:10 EDT 2020
$
Finding Linux Plugins#
So far, everything has gone smoothly, and now you might be curious about how to find the names of all the Linux plugins. There is a simple trick: run the --info command and grab the linux_ strings. There are various plugins available for different purposes. Here are some of them:
$ python2 vol.py --info | grep linux_
Volatility Foundation Volatility Framework 2.6.1
linux_apihooks - Checks for userland apihooks
linux_arp - Print the ARP table
linux_aslr_shift - Automatically detect the Linux ASLR shift
<< omitted >>
linux_banner - Prints the Linux banner information
linux_vma_cache - Gather VMAs from the vm_area_struct cache
linux_volshell - Shell in the memory image
linux_yarascan - A shell in the Linux memory image
$
Use the linux_psaux plugin to check which processes were running on the system at the time of the memory dump. Note the last command in the list: it is the insmod command you ran before the dump.
$ python2 vol.py -f ~/LiME/RHEL8.3_64bit.mem linux_psaux --profile=LinuxRedhat8_3_4_18_0-240x64
Volatility Foundation Volatility Framework 2.6.1
Pid Uid Gid Arguments
1 0 0 /usr/lib/systemd/systemd --switched-root --system --deserialize 18
2 0 0 [kthreadd]
3 0 0 [rcu_gp]
4 0 0 [rcu_par_gp]
861 0 0 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
869 0 0 /usr/bin/rhsmcertd
875 0 0 /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0 --logger=files
878 0 0 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
<< omitted >>
11064 89 89 qmgr -l -t unix -u
227148 0 0 [kworker/0:0]
227298 0 0 -bash
227374 0 0 [kworker/u2:1]
227375 0 0 [kworker/0:2]
227884 0 0 [kworker/0:3]
228573 0 0 insmod ./lime-4.18.0-240.el8.x86_64.ko path=../RHEL8.3_64bit.mem format=lime
228576 0 0
$
Want to know the network status of the system? Run the linux_netstat plugin to find the status of network connections during the memory dump:
$ python2 vol.py -f ~/LiME/RHEL8.3_64bit.mem linux_netstat --profile=LinuxRedhat8_3_4_18_0-240x64
Volatility Foundation Volatility Framework 2.6.1
UNIX 18113 systemd/1 /run/systemd/private
UNIX 11411 systemd/1 /run/systemd/notify
UNIX 11413 systemd/1 /run/systemd/cgroups-agent
UNIX 11415 systemd/1
UNIX 11416 systemd/1
<< omitted >>
$
Next, use the linux_mount plugin to see which filesystems were mounted during the memory dump:
$ python2 vol.py -f ~/LiME/RHEL8.3_64bit.mem linux_mount --profile=LinuxRedhat8_3_4_18_0-240x64
Volatility Foundation Volatility Framework 2.6.1
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec
cgroup /sys/fs/cgroup/pids cgroup rw,relatime,nosuid,nodev,noexec
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime
/dev/mapper/rhel_kvm--03--guest11-root / xfs rw,relatime
tmpfs /dev/shm tmpfs rw,nosuid,nodev
selinuxfs /sys/fs/selinux selinuxfs rw,relatime
<< omitted >>
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,relatime,nosuid,nodev,noexec
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,relatime,nosuid,nodev,noexec
bpf /sys/fs/bpf bpf rw,relatime,nosuid,nodev,noexec
cgroup /sys/fs/cgroup/memory cgroup ro,relatime,nosuid,nodev,noexec
cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,nosuid,nodev,noexec
mqueue /dev/mqueue mqueue rw,relatime
$
Curious about which kernel modules were loaded? Volatility also provides a plugin for this, linux_lsmod:
$ python2 vol.py -f ~/LiME/RHEL8.3_64bit.mem linux_lsmod --profile=LinuxRedhat8_3_4_18_0-240x64
Volatility Foundation Volatility Framework 2.6.1
ffffffffc0535040 lime 20480
ffffffffc0530540 binfmt_misc 20480
ffffffffc05e8040 sunrpc 479232
<< omitted >>
ffffffffc04f9540 nfit 65536
ffffffffc0266280 dm_mirror 28672
ffffffffc025e040 dm_region_hash 20480
ffffffffc0258180 dm_log 20480
ffffffffc024bbc0 dm_mod 151552
$
Want to know which files are opened by which processes? Use the linux_bash plugin to list this information:
$ python2 vol.py -f ~/LiME/RHEL8.3_64bit.mem linux_bash --profile=LinuxRedhat8_3_4_18_0-240x64 -v
Volatility Foundation Volatility Framework 2.6.1
Pid Name Command Time Command
-------- -------------------- ------------------------------ -------
227221 bash 2021-04-17 18:38:24 UTC+0000 lsmod
227221 bash 2021-04-17 18:38:24 UTC+0000 rm -f .log
227221 bash 2021-04-17 18:38:24 UTC+0000 ls -l /etc/zzz
227221 bash 2021-04-17 18:38:24 UTC+0000 cat ~/.vimrc
227221 bash 2021-04-17 18:38:24 UTC+0000 ls
227221 bash 2021-04-17 18:38:24 UTC+0000 cat /proc/817/cwd
227221 bash 2021-04-17 18:38:24 UTC+0000 ls -l /proc/817/cwd
227221 bash 2021-04-17 18:38:24 UTC+0000 ls /proc/817/
<< omitted >>
227298 bash 2021-04-17 18:40:30 UTC+0000 gcc prt.c
227298 bash 2021-04-17 18:40:30 UTC+0000 ls
227298 bash 2021-04-17 18:40:30 UTC+0000 ./a.out
227298 bash 2021-04-17 18:40:30 UTC+0000 vim prt.c
227298 bash 2021-04-17 18:40:30 UTC+0000 gcc prt.c
227298 bash 2021-04-17 18:40:30 UTC+0000 ./a.out
227298 bash 2021-04-17 18:40:30 UTC+0000 ls
$
Want to know which files are opened by which processes? Use the linux_lsof plugin to list this information:
$ python2 vol.py -f ~/LiME/RHEL8.3_64bit.mem linux_lsof --profile=LinuxRedhat8_3_4_18_0-240x64
Volatility Foundation Volatility Framework 2.6.1
Offset Name Pid FD Path
------------------ ------------------------------ -------- -------- ----
0xffff9c83fb1e9f40 rsyslogd 71194 0 /dev/null
0xffff9c83fb1e9f40 rsyslogd 71194 1 /dev/null
0xffff9c83fb1e9f40 rsyslogd 71194 2 /dev/null
0xffff9c83fb1e9f40 rsyslogd 71194 3 /dev/urandom
0xffff9c83fb1e9f40 rsyslogd 71194 4 socket:[83565]
0xffff9c83fb1e9f40 rsyslogd 71194 5 /var/log/messages
0xffff9c83fb1e9f40 rsyslogd 71194 6 anon_inode:[9063]
0xffff9c83fb1e9f40 rsyslogd 71194 7 /var/log/secure
<< omitted >>
0xffff9c8365761f40 insmod 228573 0 /dev/pts/0
0xffff9c8365761f40 insmod 228573 1 /dev/pts/0
0xffff9c8365761f40 insmod 228573 2 /dev/pts/0
0xffff9c8365761f40 insmod 228573 3 /root/LiME/src/lime-4.18.0-240.el8.x86_64.ko
$
Accessing the Location of Linux Plugin Scripts#
By reading the memory dump and processing this information, you can obtain even more information. If you know Python and are curious about how this information is processed, you can go to the directory where all the plugins are stored, select one that interests you, and see how Volatility obtains this information:
$ ls volatility/plugins/linux/
apihooks.py common.py kernel_opened_files.py malfind.py psaux.py
apihooks.pyc common.pyc kernel_opened_files.pyc malfind.pyc psaux.pyc
arp.py cpuinfo.py keyboard_notifiers.py mount_cache.py psenv.py
arp.pyc cpuinfo.pyc keyboard_notifiers.pyc mount_cache.pyc psenv.pyc
aslr_shift.py dentry_cache.py ld_env.py mount.py pslist_cache.py
aslr_shift.pyc dentry_cache.pyc ld_env.pyc mount.pyc pslist_cache.pyc
<< omitted >>
check_syscall_arm.py __init__.py lsmod.py proc_maps.py tty_check.py
check_syscall_arm.pyc __init__.pyc lsmod.pyc proc_maps.pyc tty_check.pyc
check_syscall.py iomem.py lsof.py proc_maps_rb.py vma_cache.py
check_syscall.pyc iomem.pyc lsof.pyc proc_maps_rb.pyc vma_cache.pyc
$
$
One reason I like Volatility is that it provides many security plugins. This information is hard to obtain manually:
linux_hidden_modules - Carves memory to find hidden kernel modules
linux_malfind - Looks for suspicious process mappings
linux_truecrypt_passphrase - Recovers cached Truecrypt passphrases
Volatility also allows you to open a shell in the memory dump, so you can run shell commands instead of all the commands above and get the same information:
$ python2 vol.py -f ~/LiME/RHEL8.3_64bit.mem linux_volshell --profile=LinuxRedhat8_3_4_18_0-240x64 -v
Volatility Foundation Volatility Framework 2.6.1
Current context: process systemd, pid=1 DTB=0x1042dc000
Welcome to volshell! Current memory image is:
file:///root/LiME/RHEL8.3_64bit.mem
To get help, type 'hh()'
>>>
>>> sc()
Current context: process systemd, pid=1 DTB=0x1042dc000
>>>
Next Steps#
Memory dumps are a great way to understand the internals of Linux. Try out all the plugins in Volatility and study their outputs in detail. Then think about how this information can help you identify intrusions or security issues. Dive deeper into how these plugins work, and even try to improve them. If you don't find a plugin for what you want to do, write one and submit it to Volatility so that others can use it.
References#
Using Open Source Tools for Linux Memory Forensics [1]
Perform Linux memory forensics with this open source tool [2]