A Snoopy How To – plus a mini explanation to linux libraries

Snoopy works for linux (it might also work for unix, im not sure). Here is watch_execve which works for FreeBSD: http://ram.kossboss.com/snoppy-freebsd-watch_execve-watch-command-executions-freebsd/


 

Snoopy logs all commands & programs that are ran by any user (or program) to a log file. This is great for auditing and finding out things. For example: you have a NAS and you run a Scrub operation from its GUI and your curious as to what commands are actually ran with that operation. It will show you exactly what commands are ran. If it doesnt show anything, most likely all of the commands were c/c++ system calls. Note that Snoopy cant pick up system and library calls via binary files, it can only pick up unix programs that are ran and their arguments (things you could run in shell like bash). So if a unix read function and write function was used to copy a file, then snoopy wont show you that, but it will show you if someone uses “cat” or “cp”.


 

* UPDATE 2015-10-21: This gets updated often. So to get the latest install instructions. Just go here: https://github.com/a2o/snoopy. Latest instructions for install are this:

First run this to get your dependencies up and running:

To install latest STABLE version:

To install latest DEV version:

NOTE: I added in –no-check-certificate so that any unix system can wget it even it doesnt have the latest ca-certs installed

NOTE: on a debian system to install the dependencies & install snoopy stable run this   apt-get update && apt-get install build-essential -y && apt-get install autoconf -y && rm -f snoopy-install.sh && wget --no-check-certificate -q -O snoopy-install.sh https://github.com/a2o/snoopy/raw/install/doc/install/bin/snoopy-install.sh && chmod 755 snoopy-install.sh && ./snoopy-install.sh stable  and if you wanted snoopy dev version run this  apt-get update && apt-get install build-essential -y && apt-get install autoconf -y && rm -f snoopy-install.sh && wget --no-check-certificate -q -O snoopy-install.sh https://github.com/a2o/snoopy/raw/install/doc/install/bin/snoopy-install.sh && chmod 755 snoopy-install.sh && ./snoopy-install.sh git-master


*  UPDATE 2014-10-24: There is now a very easy way to install snoopy and now you can activate and deactivate snoopy with a command (sweet!). Easier then all of the other ways.


 

* UPDATE 2014-10-24: word from maintainer of snoopy code – new code location: https://github.com/a2o/snoopy

git clone https://github.com/a2o/snoopy

If fail with SSL error look at this article SSL ERROR

Links below will fail, the above links will work. Still need to update scripts below to include above links.


 

So What is Snoopy?

Snoopy allows for command monitoring thru syslog (essentially every command that has been launched on the system)

The solution to my problem, see here is my problem before I found snoopy: HERE!

Snoopy, a very small and hidden program

Much smarter then my scripts in the above link

HOW IS THIS USEFUL FOR TESTING

If your using an application that has a web GUI(or QT or anyother GUI) that launches commands on the back-end of the server. Such as via tomcat or apache  (cgi) or python webserver or anythoer. First enable snoopy with snoopy-enable, then make a fake log entry logger “aaaaaaaaaaaaaaaaaaaaaaaaaaaa”, then run the gui command, and after the gui command is complete run logger “aaaaaaaaaaaaaaaaaaaaaaaaaaaa” again.

The check out your logs and you will see what the GUI did to the backend, they will be inbetween the aaaaaaaaaaaaaaaa command. Tthis will not show any C, or C++ commands, or bash embedded commands such as bg,fg,sleep, etc unless your using gnu sleep, and gnu bg, and gnu fg – not even sure if gnu bg and gnu fg exist – but for example ls can be both bash embedded or there is a version of ls that is gnu, you would see the gnu command. Also another caveat is snoopy doesnt show the pipe symbols, so when your looking at the output you need to make educated guesses to where you think they are piped. So If see for example a ps command and then next line is a grep and then an awk. Well then I can assume that ps | grep | awk was how that command went.


 

Note that now you cant install snoopy using the method below. If your curious about the installation method, just check out the snoopy-install.sh (see above on how to wget it) script. Even though the install steps below will not apply, please continue on reading and see how snoopy works at the core, you will also learn more about “librarys” – that stuff is key in linux.

Old Install Method

Quick Summary:

 

Made quick summary because this article is so long, didnt want it to bee too tl;dr.

 

Here is how snoopy works:

A program needed to be inveted that recorded when other programs run, but how do you do that? well make sure it ALWAYS runs before your intended program runs, well how do you do that? by preloading libraries (libraries are mini programs, but in reality are tool belts that can do things, so yah they are like mini programs, that can either be compiled and shoved into the final program/executable of your choice, or shared with other final programs/executables of your choice – more on this below)

Whenever a program runs in linux, it checks the libraries which the program needs and loads them, it also checks the file /etc/ld.so.preload for any libraries and preloads them. NOTE: ld.so.preload is a text file that is line delimited, every new line is a library that should be loaded before a program is ran.

A good article on /etc/ld.so.preload: http://minipli.wordpress.com/2009/07/17/ld_preload-vs-etcld-so-preload/

Anything in /etc/ld.so.preload gets preloaded for EVERY program. So the library which gets called is the snoopy library/program. That library intercepts the command name and who ran it and appends that information to syslog.

The result is command monitoring just as I wanted it

Pre install and download:

To install it, first prepare your compiling environment (you will need to have things installed like make and gcc, hopefully you already have them):

 

 

NOTE: when running git clone it will download the source code into a folder name that they chose, in this case they chose code. You can just run ls -lisah to see whats the name of the folder that git clone downloaded

OPTIONAL STEP BEFORE INSTALLING: Allow for snoopy to record bigger arguments in commands. change the snoopy.h file so that the snoopy installation allows for recording bigger arguments:

Find this line at the top:

change to 3 times the size (thanks to video by Kelvin Tan Thiam Teck at this link http://www.youtube.com/watch?v=Lts34JkmwuU)

 

Or with a oneliner using sed you can replace it like this:

 

We basically set the global variable which sets the max size of arguments snoopy records.

Back to install:

BEFORE CONFIGURE (you will notice that this git clone doesnt have the configure script, so you can make it with autoheader and autoconf – dont worry I got this from the README file which tells you exactly what to do):

NOTE: if running solaris or opensolaris, the make should fail, you will need to run a fix, that fix is detailed out at top in the summary

Now the software has put its library in the OS library directories. Now its up to you to enable the library (enable in snoopys terms meaning, tell /etc/ld.so.preload about this snoopy lib, so that the snoopy lib gets loaded before every program). Test if for your self do a test of: # cat /etc/ld.so.preload, # make enable, and cat it again # cat /etc/ld.so.preload

ONLY READ THIS NEXT PARAGRAPH IF “make enable” FAILED: if the make enable fails (confirm it failed by running cat /etc/ld.so.preload , if you dont see anything there most likely it failed, also if you dont see the path where snoopy.so was isntalled to then it also failed), just run it manually, you will see what it tried to run its something like ./enable.sh /usr/local/lib . Dont take my word for it, on your system it could be something other then /usr/local/lib. Its just going to the folder where snoopy.so was installed. You can find the full path and just truncate (or keep everything) to the left of the last /, using this command:  find / -name "snoopy.so" . After getting the folder just run the above command  ./enable.sh <folder where snoopy.so is at> , replace <folder where snoopy.so is at> with the actual folder, not including the angle brackets (sorry I have to be so descriptive, thats why all my stuff is so tl;dr). Also instead of running ./enable.sh, you can just manually do what it was going to do. Just edit or create (if its missing) the file called /etc/ld.so.preload. Here is the command to manually do what make enable would do, if your snoopy.so is in /usr/local/lib:  echo "/usr/local/lib/snoopy.so" >> /etc/ld.so.preload .  Obviously replace /usr/local/lib/snoopy.so with the full-file name (folder name included)  – aka: absolute path – – where snoopy.so installed for you.

Thats it its now enabled. Skip to section “how to use” if you dont want to get in the details of how this snoopy is installed (i covered some of it above already).

So you can see that “make enable” writes to “/etc/ld.so.preload” one line with the location of the snoopy lib.

My ld.so.preload looks like this (and yours will probably too because the source code compilation with ‘configure’ and ‘make’ and ‘make install’ put it there):

To test to see that /etc/ld.so.preload gets checked out before any program try this: “# apt-get install strace” now do an strace (monitors lib and syscalls) on any program “# strace touch test123” “strace rm test123”. Notice that in the strace you see

Then later you see:

 

How to use:

Now that snoopy is enabled your done, you can now just delete the source directory, and begin running commands.

To see the commands, we need to look thru where syslog saves its output

or if you have nested log folders do a recursive search

Notice where all of the snoopy output goes

For me all of the latest commands were being written to auth.log

So for me to follow the latest commands:

NOTE: example output can be seen below

More in depth – How to use:

Sorry this whole next part is in code blocks, everything I wanted to say is in comments after # lines.

 

 

If not working:

Restart your shell or reboot:

If you cant see anything being recorded.

if your in ssh, restart ssh, dont worry you connection should maintain if your ssh is smart
if sysvinit/init system:

if systemctl:

or

or

Simply log out and log back in

Also try rebooting the system, thats the perfect test. The test even more perfect then a reboot? Reboot again, then if the program works without problems after 2 reboots your sure as @#$@ know it works.

Make sure programs link to snoopy.so library:

Mini intro to libraries in linux:

static libraries are extra tools for your program to use which end with .a extension and they are compiled into the program making them bigger (but the tools / the library always comes embedded into the final program thus you dont need to install any shared libs as a prequiste.) PRO: comes with app, no preqs, CON: bigger footprint. The library and the program sit together as 1 file in the end (you still have the original shared lib file sitting in your lib folder somewhere, so that it can be used by other program)

dynamic libraries are with the extension .so (plus maybe extra numbers after like .so.1) then are not compiled into the program but they are linked to the program. if 10 programs are using the same 11K library, that library will only take up 11K of memory. However if it was a shared library then the footprint would be 11K for every lib times 10 apps using it (using a seperate version of it, that sits compiled into each program) so it would be a 110k total foot print (10 times bigger then if it were a dynamic library). the disadvantage of course. PRO: smaller footprint and shared, CON: need to have it in the system as an extra file

ldd command can be used to see which librarys (its like windows dlls, they are not programs but contain extra functions and methods that a program can use, “my analogy to what libraries and dll files  are like:hey you dont want to retype an ‘cos’ and ‘sin’ function everytime you write a program that uses basic math) are dynamically linked(also known as shared librarys) thus a single library can be used by multiple running programs at once (so the same lib doesn’t have to load as many times as there are programs using it, this saves memory). A static library compiles the librarys into the executable (so they dont require library prerequisites, as they are compiled into the program). ldd shows you if a program is dynmically links/shared by showing which libraries it uses (you can see snoopy being called right after the vdso library-sidenote: vdso allows for faster cpu interrupts by using SYSENTER and SYSEXIT instead of intruppts – this was important when cpus became faster and intruppts bottleneck performance), ldd shows you if its a static library by telling you that the program has no dynamic links (no shared libraries associated to it). So we see he snoopy although is dynamically linked to ln and touch will show when those commands run. Also even though we see that zcat is static and doesn’t mention snoopy, it still runs snoopy so zcat is recorded into syslog, but how its not linked? well my guess is ld.so.preloads work by linking thier uses on top of programs whether they are static linked or shared link, that way you ensure consitency with all apps running libs in ld.so.preloads. BONUS TIP: if a program is compiled with static, then all of the libraries are in the final executable file so the program will be bigger, but can run without prerequesite libraries laying around in your system

 OUTPUT FROM BOTH PROOVING THAT BOTH TYPES OF DYNMICALLY AND STATIC LINKED APPS GET RECORDED:

Example output of snoopy logging in auth

NOTE: for me and my default rsyslog/syslong config, all of the snoopy output goes to auth.log

To uninstall:

1. just remove the reference line in /etc/ld.so.preload

Or manually go in

Now delete the line that mentions snoopy.so

2. Now delete the /usr/local/lib/snoopy.so

If yours is else where and you cant remember where

To delete what it find

or easier but more confidently:

3. Also unset any environmental variable that references snoopy (LD_PRELOAD, LD_PRELOAD_32 and LD_PRELOAD_64).

Caveats and limitations of snoopy

1- I found one problem/limitation with snoopy is that it wont capture a full command if your arguments are bigger then 4096 characters (or if you changed the gloval variable in snoopy.h before compilation then you can increase it to like 3 times the amount of charcters – so that limitation isnt really big)

2- The big limitation is when you pipe commands, it captures each command and its arguments seperately

each command shows up seperately in the final snoopy syslog capture.

 

One thought on “Snoopy – tutorial – the command logger for any user – also mini explanation to linux libraries

Leave a Reply

Your email address will not be published. Required fields are marked *