Passive DNS collection and analysis using YaF and Mediator

Passive DNS is a useful tool for any analysts teams toolbox, I have noted several public sensors here but they only see data (queries and responses) that transverse their sensors. I have been working on setting up passive DNS using Yet another Flowmeter (YaF) and Mediator (YaF to MySQL) to fill the gap where third-party sensors may not be providing the coverage I would like. Passive DNS can provide tremendous insight and analytics upon DNS queries that users and/or malware may beperforming. A few items of interest:

Hostnames that have a large number of IP addresses associated with them in a short time period and they have only been visited by very few hosts host on the network.
Tertiary name usage associated with a specific domain?
When was the domain first resolved on the network and further, how often is it being resolved and by whom?
A recently accessed/registered domain with short time to live (TTLs) often associated with new IP addresses may indicate malicious activity, or a CDN.
Queries for TLDs that you typically do not interact with may be worth looking into.
Users using non-approved DNS servers

Passive DNS may be also helpful in tracking infections using Fast-fluxwhich make blocking the C2 difficult as the attackers will create algorithms to rotate the IP addresses and even the hostnames in the case of double-flux. (TorPig) The list goes on but in a nutshell, I wanted to be able to perform this activity without having to rely on having all of the DNS server logs in a centralized location, especially since users may reconfigure their DNS settings to use non-approved servers, e.g. BYOD.

This entry demonstrates how to build and setup YaF and Mediator both of which are available from the CERT NetSA site and should be considered complementary to the documentation the NetSA team have already provided for each of the respective tools. This setup was tested on CentOS 6.4 but most Linux distributions should work fine.

Have site reconfigure interfaces on all hosts. eth0 should be management interface and eth1 should be the tap OR whatever makes sense, this need to happen every time the host comes up, i.e.

sudo ifconfig eth1 up promisc

Ensure development libraries/dependencies are installed. Some may require enabling the optional software channel

sudo yum install glib2-devel lzo  gcc-c++ libpcap-devel pcre-devel

Install libfixbuf

cd libfixbuf-1.3.0
./configure
make
sudo make install

Install YaF

cd yaf-2.3.3
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
./configure --with-libpcap --enable-applabel --enable-plugins
make
sudo make install

Edit ld

sudo echo "/usr/local/lib" >> /etc/ld.so.conf
sudo /sbin/ldconfig
sudo /sbin/ldconfig -v | grep libzmq # should rebuild the cache including zmq too.

export PATH=$PATH:/usr/local/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

Configure cmake

cd cmake-2.8.10.2
./configure
gmake

Optionally, configure YaF to File output for testing purposes.

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
cd yaf_file_mediator-1.1.0/
./configure
../cmake-2.8.10.2/bin/cmake .
make

Configure YaF to MySQL

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
cd yaf_silk_mysql_mediator-1.4.0
../cmake-2.8.10.2/bin/cmake .
./configure --with-mysql
make

Next, populate create a database and respective tables:

./yafMySQL -o localhost -n username -p password -d eflows

Setup YaF to start capturing. Here we are only capture DNS traffic and rotating the files written to disk after 5 minutes. Originally set to 10 minutes but yaf_silk_mysql_mediator would segmentation fault because MySQL would close the connection before all of the data would insert. We have a continuous method that works a little better which we should a little later. We lock the file so that another process cannot take the file that is currently being written to.

sudo /usr/local/bin/yaf --live pcap --in eth1 --out /data/ipfix/ --rotate 600 --filter="port 53" --applabel --applabel-rules=/usr/local/etc/yafApplabelRules.conf --max-payload=1000 --plugin-name=/usr/local/lib/yaf/dpacketplugin.la --plugin-opts="53" --lock --become-user=nobody --become-group=nobody &

Testing the output of a YaF

yaf_file_mediator-1.1.0/yaf_file_mediator --input /data/ipfix/filename.yaf --output test.txt

After a few minutes, you should be able to parse the filename.yaf that was first written (in this case 5 minutes). The contents of test.txt should be similar to the following:

-------------------------------
Template ID is 45841
Application Label: 53
Source IP: 192.168.0.5
Destination IP: 8.8.8.8
Source Port: 53855
Dest Port: 53
Flow Attributes: 1
Rev Flow Attributes: 0
flowStartTime: 2013-04-24 23:53:43
flowEndTime: 2013-04-24 23:58:02
flowEndReason: 1
Protocol: 17
Octet Total Count: 120
Rev Octet count: 244
Packet Total Count: 2
Rev Packet Total Count: 2
DNS ID: 32852 Type: 28 RR Section: 0 TTL: 0 Query: www.google.com.
DNS ID: 32852 Type: 28 RR Section: 1 TTL: 204 RRName: www.google.com. AAAA: 2607:f8b0:400c:0c04::0069

-------------------------------
Template ID is 45841
Application Label: 53
Source IP: 192.168.0.5
Destination IP: 8.8.8.8
Source Port: 50845
Dest Port: 53
flowStartTime: 2013-04-24 23:58:02
flowEndTime: 2013-04-24 23:58:02
flowEndReason: 1
Protocol: 17
Octet Total Count: 60
Rev Octet count: 156
Packet Total Count: 1
Rev Packet Total Count: 1
DNS ID: 21141 Type: 1 RR Section: 0 TTL: 0 Query: www.google.com.
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.103
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.99
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.105
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.104
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.106
DNS ID: 21141 Type: 1 RR Section: 1 TTL: 208 RRName: www.google.com. A: 74.125.26.147

After you have confirmed that your YaF entries contain records, adda little automation. This will scoop up the files in the directory where the YaF files are being written, place them in the MySQL DBMS and delete the file. Note, if you start seeing “Segmentation Fault” then MySQL maybe closing the connection before all of the records from the YaF file could be written to the DBMS. You can try modifying MySQL parameters or reduce the the size of YaF files being written to disk in order to try mitigating this symptom if it occurs in your environment.

for i in $( ls /data/ipfix/*.yaf ); do /home/user/silk-installs/yaf_silk_mysql_mediator-1.4.0/yaf_silk_mysql_mediator --in-file $i --mysql-host localhost --name username --pass password --database eflows && sudo rm $i; done

Here is our first query, lets see who has recently made requests for www.google.com.

mysql> SELECT rrname,rrval,srcip4,dstip4,flowStartMilliseconds FROM dns d, flows f WHERE f.id = d.id AND rrname LIKE "www.google.com." GROUP by rrval ORDER BY f.id DESC LIMIT 50;
+-----------------+---------------------------+------------+-----------+-----------------------+
| rrname          | rrval                     | srcip4     | dstip4    | flowStartMilliseconds |
+-----------------+---------------------------+------------+-----------+-----------------------+
| www.google.com. | 2001:4860:4001:0802::1012 | 3232235525 | 134744072 | 2013-05-03 17:47:24   |
| www.google.com. | 2001:4860:4001:0801::1014 | 3232235525 | 134744072 | 2013-05-03 15:35:32   |
| www.google.com. | 2001:4860:4001:0802::1014 | 3232235525 | 134744072 | 2013-05-03 11:28:42   |
| www.google.com. | 2001:4860:4001:0801::1010 | 3232235525 | 134744072 | 2013-05-02 16:48:31   |
| www.google.com. | 2001:4860:4001:0802::1011 | 3232235525 | 134744072 | 2013-05-02 13:33:57   |
| www.google.com. | 2001:4860:4001:0803::1010 | 3232235525 | 134744072 | 2013-05-02 12:01:56   |
| www.google.com. | 2607:f8b0:4004:0801::1012 | 3232235525 | 134744072 | 2013-05-01 21:36:55   |
| www.google.com. | 2001:4860:4001:0802::1010 | 3232235525 | 134744072 | 2013-05-01 12:44:52   |
| www.google.com. | 74.125.239.80             | 3232235525 | 134744072 | 2013-05-01 10:45:04   |
| www.google.com. | 74.125.239.83             | 3232235525 | 134744072 | 2013-05-01 10:45:04   |
| www.google.com. | 74.125.239.82             | 3232235525 | 134744072 | 2013-05-01 10:45:04   |
| www.google.com. | 74.125.239.81             | 3232235525 | 134744072 | 2013-05-01 10:45:04   |
| www.google.com. | 74.125.239.84             | 3232235525 | 134744072 | 2013-05-01 10:45:04   |
| www.google.com. | 2607:f8b0:4004:0802::1010 | 3232235525 | 134744072 | 2013-04-29 19:54:00   |
| www.google.com. | 2607:f8b0:4005:0802::1010 | 3232235525 | 134744072 | 2013-04-28 15:52:00   |
| www.google.com. | 2607:f8b0:4004:0803::1013 | 3232235525 | 134744072 | 2013-04-28 15:05:53   |
| www.google.com. | 2607:f8b0:4005:0802::1011 | 3232235525 | 134744072 | 2013-04-27 14:45:35   |
| www.google.com. | 2607:f8b0:4004:0801::1013 | 3232235525 | 134744072 | 2013-04-26 18:53:45   |
| www.google.com. | 2607:f8b0:4005:0802::1012 | 3232235525 | 134744072 | 2013-04-26 13:55:51   |
| www.google.com. | 2607:f8b0:4005:0802::1013 | 3232235525 | 134744072 | 2013-04-26 12:35:18   |
| www.google.com. | 74.125.239.145            | 3232235525 | 134744072 | 2013-04-26 12:03:10   |
| www.google.com. | 74.125.239.148            | 3232235525 | 134744072 | 2013-04-26 12:03:10   |
| www.google.com. | 74.125.239.146            | 3232235525 | 134744072 | 2013-04-26 12:03:10   |
| www.google.com. | 74.125.239.147            | 3232235525 | 134744072 | 2013-04-26 12:03:10   |
| www.google.com. | 74.125.239.144            | 3232235525 | 134744072 | 2013-04-26 12:03:10   |
| www.google.com. | 2607:f8b0:4005:0802::1014 | 3232235525 | 134744072 | 2013-04-26 11:31:59   |
| www.google.com. | 74.125.228.112            | 3232235525 | 134744072 | 2013-04-25 16:25:39   |
| www.google.com. | 74.125.228.114            | 3232235525 | 134744072 | 2013-04-25 16:25:39   |
| www.google.com. | 74.125.228.113            | 3232235525 | 134744072 | 2013-04-25 16:25:39   |
| www.google.com. | 74.125.228.115            | 3232235525 | 134744072 | 2013-04-25 16:25:39   |
| www.google.com. | 74.125.228.116            | 3232235525 | 134744072 | 2013-04-25 16:25:39   |
| www.google.com. | 2607:f8b0:4004:0802::1012 | 3232235525 | 134744072 | 2013-04-25 11:29:45   |
| www.google.com. | 2607:f8b0:4004:0803::1014 | 3232235525 | 134744072 | 2013-04-24 20:33:42   |
| www.google.com. | 2607:f8b0:400e:0c04::006a | 3232235525 | 134744072 | 2013-04-24 18:04:19   |
| www.google.com. | 2607:f8b0:400e:0c02::006a | 3232235525 | 134744072 | 2013-04-24 15:26:22   |
| www.google.com. | 74.125.228.20             | 3232235525 | 134744072 | 2013-04-24 12:05:43   |
| www.google.com. | 74.125.228.16             | 3232235525 | 134744072 | 2013-04-24 12:05:43   |
| www.google.com. | 74.125.228.18             | 3232235525 | 134744072 | 2013-04-24 12:05:43   |
| www.google.com. | 74.125.228.19             | 3232235525 | 134744072 | 2013-04-24 12:05:43   |
| www.google.com. | 74.125.228.17             | 3232235525 | 134744072 | 2013-04-24 12:05:43   |
| www.google.com. | 2607:f8b0:4004:0801::1014 | 3232235525 | 134744072 | 2013-04-23 20:43:26   |
| www.google.com. | 74.125.228.50             | 3232235525 | 134744072 | 2013-04-23 20:38:43   |
| www.google.com. | 74.125.228.51             | 3232235525 | 134744072 | 2013-04-23 20:38:43   |
| www.google.com. | 74.125.228.52             | 3232235525 | 134744072 | 2013-04-23 20:38:43   |
| www.google.com. | 74.125.228.48             | 3232235525 | 134744072 | 2013-04-23 20:38:43   |
| www.google.com. | 74.125.228.49             | 3232235525 | 134744072 | 2013-04-23 20:38:43   |
| www.google.com. | 2607:f8b0:4004:0801::1011 | 3232235525 | 134744072 | 2013-04-23 18:38:52   |
| www.google.com. | 2607:f8b0:400e:0c01::0067 | 3232235525 | 134744072 | 2013-04-23 15:57:45   |
| www.google.com. | 2607:f8b0:4004:0801::1010 | 3232235525 | 134744072 | 2013-04-23 15:07:59   |
| www.google.com. | 2607:f8b0:400e:0c01::0069 | 3232235525 | 134744072 | 2013-04-23 12:30:28   |
+-----------------+---------------------------+------------+-----------+-----------------------+

Here is a similar query but we want to see any tertiary youtube.com domains and sort by the lookup returned.

mysql> SELECT qr,type,auth,nx,ttl,rrname,rrval from dns WHERE rrname LIKE "%.youtube.com." GROUP BY rrval LIMIT 50;
+------+------+------+------+------+--------------------------------+----------------+
| qr   | type | auth | nx   | ttl  | rrname                         | rrval          |
+------+------+------+------+------+--------------------------------+----------------+
|    0 |    1 |    0 |    0 |    0 | www.youtube.com.               |                |
|    1 |    1 |    0 |    0 |  300 | v17.lscache2.c.youtube.com.    | 12.216.80.12   |
|    1 |    1 |    0 |    0 | 1800 | r2.sn-5uu-vgqe.c.youtube.com.  | 12.216.80.13   |
|    1 |    1 |    0 |    0 | 1800 | r3.sn-5uu-vgqe.c.youtube.com.  | 12.216.80.14   |
|    1 |    1 |    0 |    0 | 1800 | r4.att-ord1.c.youtube.com.     | 12.216.80.15   |
|    1 |    1 |    0 |    0 | 1800 | r6.sn-5uu-vgqe.c.youtube.com.  | 12.216.80.17   |
|    1 |    1 |    0 |    0 | 1714 | r8.sn-5uu-vgqe.c.youtube.com.  | 12.216.80.19   |
|    1 |    1 |    0 |    0 | 1741 | r1.sn-5uu-vgql.c.youtube.com.  | 12.216.80.44   |
|    1 |    1 |    0 |    0 | 1800 | r2.sn-5uu-vgql.c.youtube.com.  | 12.216.80.45   |
|    1 |    1 |    0 |    0 | 1800 | r3.sn-5uu-vgql.c.youtube.com.  | 12.216.80.46   |
|    1 |    1 |    0 |    0 | 1279 | r4.sn-5uu-vgql.c.youtube.com.  | 12.216.80.47   |
|    1 |    1 |    0 |    0 | 1800 | r6.sn-5uu-vgql.c.youtube.com.  | 12.216.80.49   |
|    1 |    1 |    0 |    0 | 1800 | r7.sn-5uu-vgql.c.youtube.com.  | 12.216.80.50   |
|    1 |    1 |    0 |    0 | 1739 | r8.sn-5uu-vgql.c.youtube.com.  | 12.216.80.51   |
|    1 |    1 |    0 |    0 | 1800 | r12.sn-hp576nes.c.youtube.com. | 173.194.17.17  |
|    1 |    1 |    0 |    0 | 1800 | r20.sn-hp576nes.c.youtube.com. | 173.194.17.25  |
|    1 |    1 |    0 |    0 | 1800 | r6.sn-q4f7dnel.c.youtube.com.  | 173.194.24.11  |
|    1 |    1 |    0 |    0 | 1800 | r1.dfw06s08.c.youtube.com.     | 173.194.24.134 |
|    1 |    1 |    0 |    0 | 1800 | r15.sn-q4f7dn7r.c.youtube.com. | 173.194.24.148 |
|    1 |    1 |    0 |    0 | 1800 | r18.sn-hp576n7d.c.youtube.com. | 173.194.29.119 |
|    1 |    1 |    0 |    0 | 1800 | r9.sn-hp576n7z.c.youtube.com.  | 173.194.29.46  |
|    1 |    1 |    0 |    0 | 1800 | r5.sn-ab5e6ner.c.youtube.com.  | 173.194.31.10  |
|    1 |    1 |    0 |    0 | 1800 | r1.sn-ab5e6nle.c.youtube.com.  | 173.194.31.102 |
|    1 |    1 |    0 |    0 |  640 | r2.sn-ab5e6nle.c.youtube.com.  | 173.194.31.103 |
|    1 |    1 |    0 |    0 | 1800 | r3.sn-ab5e6nle.c.youtube.com.  | 173.194.31.104 |
|    1 |    1 |    0 |    0 | 1800 | r4.sn-ab5e6nle.c.youtube.com.  | 173.194.31.105 |
|    1 |    1 |    0 |    0 | 1800 | r5.sn-ab5e6nle.c.youtube.com.  | 173.194.31.106 |
|    1 |    1 |    0 |    0 | 1800 | r6.sn-ab5e6nle.c.youtube.com.  | 173.194.31.107 |
|    1 |    1 |    0 |    0 | 1800 | r7.sn-ab5e6nle.c.youtube.com.  | 173.194.31.108 |
|    1 |    1 |    0 |    0 | 1800 | r8.sn-ab5e6nle.c.youtube.com.  | 173.194.31.109 |
|    1 |    1 |    0 |    0 |  705 | r6.sn-ab5e6ner.c.youtube.com.  | 173.194.31.11  |
|    1 |    1 |    0 |    0 | 1800 | r9.sn-ab5e6nle.c.youtube.com.  | 173.194.31.110 |
|    1 |    1 |    0 |    0 | 1800 | r10.sn-ab5e6nle.c.youtube.com. | 173.194.31.111 |
|    1 |    1 |    0 |    0 |  292 | r11.sn-ab5e6nle.c.youtube.com. | 173.194.31.112 |
|    1 |    1 |    0 |    0 | 1800 | r12.sn-ab5e6nle.c.youtube.com. | 173.194.31.113 |
|    1 |    1 |    0 |    0 |  178 | r13.sn-ab5e6nle.c.youtube.com. | 173.194.31.114 |
|    1 |    1 |    0 |    0 | 1800 | r14.sn-ab5e6nle.c.youtube.com. | 173.194.31.115 |
|    1 |    1 |    0 |    0 | 1800 | r15.sn-ab5e6nle.c.youtube.com. | 173.194.31.116 |
|    1 |    1 |    0 |    0 | 1800 | r16.sn-ab5e6nle.c.youtube.com. | 173.194.31.117 |
|    1 |    1 |    0 |    0 | 1800 | r17.sn-ab5e6nle.c.youtube.com. | 173.194.31.118 |
|    1 |    1 |    0 |    0 | 1800 | r18.sn-ab5e6nle.c.youtube.com. | 173.194.31.119 |
|    1 |    1 |    0 |    0 | 1653 | r7.sn-ab5e6ner.c.youtube.com.  | 173.194.31.12  |
|    1 |    1 |    0 |    0 | 1800 | r19.sn-ab5e6nle.c.youtube.com. | 173.194.31.120 |
|    1 |    1 |    0 |    0 | 1800 | r20.sn-ab5e6nle.c.youtube.com. | 173.194.31.121 |
|    1 |    1 |    0 |    0 | 1800 | r8.sn-ab5e6ner.c.youtube.com.  | 173.194.31.13  |
|    1 |    1 |    0 |    0 |   81 | r1.sn-ab5e6nll.c.youtube.com.  | 173.194.31.134 |
|    1 |    1 |    0 |    0 | 1800 | r2.sn-ab5e6nll.c.youtube.com.  | 173.194.31.135 |
|    1 |    1 |    0 |    0 | 1800 | r3.sn-ab5e6nll.c.youtube.com.  | 173.194.31.136 |
|    1 |    1 |    0 |    0 | 1800 | r4.sn-ab5e6nll.c.youtube.com.  | 173.194.31.137 |
|    1 |    1 |    0 |    0 | 1800 | r5.lga15s22.c.youtube.com.     | 173.194.31.138 |
+------+------+------+------+------+--------------------------------+----------------+
50 rows in set (22.17 sec)

An alternative method is to write YaF records directly to mediator, and further the MySQL DBMS rather then writing files to disk although youcan still do this with the appropriate toggles. Here is example usage to start the processes:

$ ./silk-installs/yaf_silk_mysql_mediator-1.4.0/yaf_silk_mysql_mediator --in-host=127.0.0.1 --in-port=18000 --mysql-host=localhost --name=username --pass password --database eflows
$ sudo /usr/local/bin/yaf --live pcap --in eth1 --out 127.0.0.1 --ipfix-port=18000 --ipfix tcp --log=/var/log/yaf.log --filter="port 53" --applabel --applabel-rules=/usr/local/etc/yafApplabelRules.conf --max-payload=1000 --plugin-name=/usr/local/lib/yaf/dpacketplugin.la --plugin-opts="53" &

Ensure YaF and mediator are connected:

$ sudo netstat -tupan|grep yaf
tcp        0      0 127.0.0.1:18000             0.0.0.0:*                   LISTEN      6497/yaf_silk_mysql
tcp        0      0 127.0.0.1:47417             127.0.0.1:18000             ESTABLISHED 6513/yaf
tcp        0      0 127.0.0.1:18000             127.0.0.1:47417             ESTABLISHED 6497/yaf_silk_mysql

You may use the following MySQL query to see when the table was last updated to ensure records are being inserted on a regular basis:

mysql> SHOW TABLE STATUS in eflows;

After a few minutes of collection, query a domain that has been recently resolved and you should see it in the DBMS.

mysql> SELECT rrname,rrval from dns WHERE rrname LIKE "%rsreese.com." GROUP BY rrval LIMIT 10;
+--------------+--------------------------------+
| rrname       | rrval                          |
+--------------+--------------------------------+
| rsreese.com. |                                |
| rsreese.com. | 2600:3c02::f03c:91ff:fe96:f7bd |
| rsreese.com. | 74.207.234.79                  |
| rsreese.com. | ns1.linode.com.                |
| rsreese.com. | ns2.linode.com.                |
| rsreese.com. | ns3.linode.com.                |
| rsreese.com. | ns4.linode.com.                |
| rsreese.com. | ns5.linode.com.                |
+--------------+--------------------------------+
8 rows in set (18.26 sec)

There are a number of different fields available for query so I leave it to you to come up with whatever is most useful for you. Further, think of how you could write a shiny front-end for analysts to use rather then having to use the MySQL command line interface. Hope you found this useful and leave a comment if you did or have any questions.

Stephen Reese

Passive DNS collection and analysis using YaF and Mediator

Comments