I. What is Cheap Logger?

The Cheap Logger is an attempt to create a system capable of logging high-speed internet traffic using off-the-shelf equipment rather than large, specially designed systems.

II. What hardware comprises a Cheap Logger system?

The basic hardware setup of a Cheap Logger system consists of a server with at least two high-speed ethernet ports, an ethernet switch, and a scalable number of client systems. A tap from the network to be logged is connected to one of the high-speed ethernet ports on the server, while the other port connects the server to the switch. A third ethernet port is recommended on the server for remote controlling of the system if you do not desire to have your controlling traffic itself logged. The client systems are all connected to ports on the switch. The server should be a system capable of keeping up with the incoming network traffic, while the client systems need only possess sufficient storage space to save the traffic to disk.

III. What software controls a Cheap Logger?

The Cheap Logger software suite consists of three main components. The first is the server component, which is called Ethermux. This program runs on the server and is charged with the task of reading incoming packets as quickly as it can and routing them round-robin to the clients connected on the switch. The client program, dubbed Logger, runs on each of the client machines and dumps the packets that it receives to disk. The final main component is the post processor, which must be run once a logging session has been completed in order to sort the logged data by te time it was received by the server.

IV. How does a Cheap Logger system log traffic?

When all the hardware is connected, the logger is started on all of the clients that will be used, and Ethermux is started on the server. Ethermux will detect the clients and add them to its client list. Ethermux then uses libpcap in promiscuous mode to capture all traffic on the inbound port and throws the packets into a buffer. Another thread stamps an ID onto each packet in the order that the packets were received, and the exact time that the packets were received is stored in another buffer. The captured packets are then sent to the outbound port; each packet is destined for a single client, which is chosen in round-robin order. In this way, packets on an incoming gigabit connection can be dispersed to different machines at a speed low enough for their hard drives to keep up with the traffic stream. Clients are designed to be hot-swappable during the capturing phase, so that, for instance, if a client runs out of disk space, it can notify the server and then be replaced during a capture session.

I. The Web Status component

My work this sumer started with an addition to the Cheap Logger system. The goal of the Web Status component is to display the status of the system elsewhere on the planet, such as on a web site. The component consists of two parts. The first is the status server, which runs as a thread inside of the Ethermux program. It opens a UDP port that Web Status Collectors may connect to. This thread has access to the statistics variables, and periodically checks for status requests. Upon receipt of a request, the server thread sends the statistics to the Collector. A Status Collector is a standalone program that can be run as a daemon on another machine, such as a web server. The collector is set to request a given Ethermux server for updated status information every second or two (customizable). When the collector receives information, it writes it to a file which can be read by other applications. The way that we have used this system so far has been to run an Ethermux server on our Cheap Logger system, run the collector daemon on a web server, and use a PHP page on the server to show visitors the status of the Ethermux server. In this way, anyone who can view the web site can view the status of the Cheap Logger system without requiring access to the system itself. The server component was designed such that it does not present a security risk to Ethermux itself. It accepts a maximum of one request each second (customizable), can be set to only allow requests from a specified address, can be disabled, and only accepts a single type of command from collectors, namely the “GET” command. The Web Status Component was designed to be versatile and can be easily modified for use with other projects. Once the component is initialized, the project need only use a printf-like function to write status information to the collector.

II. Timestamp Trouble

The original ethermux code worked, but had a few problems in it. The code that dealt with timestamp-logging was designed to be highly efficient, but unfortunately it would fail every once in a while. I rewrote this portion of the code in a way that is less efficient in terms of space usage (which uses more memory and more bandwidth), but is much easier to understand and not likely to be corrupted. Also, the way in which timestamps were to be matched with their respective packets was by hashing certain bytes from the packet to form a ID that is 99+% likely to be unique in a set of ten or so packets. This works most of the time, but can cause problems and increase complexity when the hash fails to be unique. The timestamp packet format could still stand for some improvement, as timestamp bandwidth usage is non-negligible at high capture speeds. A more space-efficient way to store stamps should be used, but it must be a scheme that works 100% of the time.

III. Modularize the Beast

When I began work on the ethermux server, the majority of the code was contained in a single source file. I tried to make it easier to deal with by splitting this file into a number of smaller ones which group related functions, such as those dealing with the client, statistics, threads, and timestamps.

IV. New Logger Client

The biggest item that I tackled this summer was the logger client. The original client worked, but it showed no status output besides the names of the logfiles that it created and, more importantly, would choke when receiving data at a high rate of speed. It worked by spawning a new dumper thread whenever a packet queue becomes full. This thread then began dumping the packets to disk. In the meantime, a new queue was created to catch new packets. When the client received data faster than it could handle, it would spawn a large number of threads. By looking at the output of the top command while the logger ran, one could see that it steadily used more and more virtual memory. Also, when the packet stream ends, the top command shows that not all of the spawned threads finish up and exit. I decided to redesign the logger and to start from scratch. I wrote the new logger in C (previously, the entire project was written in C besides the C++-based logger). The new logger works by first creating a large, fixed-size array to hold packets, and a small array of pcap dump pointers (essentially file pointers). It then spawns two main threads: The first, the Dumper Thread, is in charge of dumping packets in the array to the current dump file. The second is the Dump Cycler thread, which monitors the dumper array to ensure that all of the dumper pointers are open and ready to be written to. Only one of the dumpers is the active dumper, which the Dumper Thread writes to. The Dumper Thread continues writing buffered packets to disk until it fills the file to its maximum size (10MB by default). The Dumper then closes the active dumper and cycles it to the next available dumper. Meanwhile, the Dump Cycler periodically checks to see if any of the dumpers have been closed. When it finds a NULL dumper, it opens a new dump file. In this way, there should always be an active dumper that is ready to receive packets. Also, the number of threads is limited to a fixed number. When running the new logger alongside a top command, top reveals that the new logger uses more CPU power, but only uses a fixed number of threads and a fixed amount of memory – an amount that is by far lower than the original logger's ever-increasing usage, even a short time period after the logging begins. Also, I ran tests in which ethermux ran using three clients, two of which were new clients and one of which was an original client. The input data was above 500Mbps, a rate sufficiently high to ensure that the loggers would have to drop a large number of packets. At the end, both new clients had received nearly the same amount of data – approximately 40MB. The original logger lagged behind at just over 30MB. The new logger is not yet polished and could be improved, but at this point it seems to work much better than the original.

V. Testing

My final contribution to the project is a small testing program which generates test traffic. Given a packet size, an amount of time to sleep between packet transmissions, and the number of packets to send, the dumpgen program will produce a tcpdump file that can be sent to ethermux's incoming port to test the system's capabilities. It makes generic IP packets which have the IPv4 source address field filled with sequential numbers counting upwards from zero. Thus, the Cheap Logger packet logs can be examined afterwards to verify the total number of packets that were successfully logged and which ones were dropped. It also makes it easy to see if the post processor does it's job correctly when it tries to put the packets back in the order that they were received in. I also wrote two small programs to verify that all of the dumpgen traffic sent by tcpreplay makes it to the clients. The program 'checklogs' looks at the log files on a single client and compiles a list of all of the dumpgen packet IDs. The program 'showdropped', when given a list of checklogs' output files for multiple clients, merges and sorts them into a single file and tells how many packets were dropped and which IDs they had. This small testing suite is useful for testing the limits of the Cheap Logger system.

I. Required Hardware

Although one could run a session with a single computer with a loopback interface, this defeats the purpose of the system. In order to benefit from the design of the cheap logger, at least two computers are required, and more can be added to accomadate faster capture speeds. Note that because packets are sent to the clients round-robbin, the capture goodness is only as great as the slowest client. Another important matter is that the clients must possess enough hard disk space for the capture. On average, clients will receive very similar amounts of data. All machines should be running a Unix-like operating system. The software should work with Linux, OS X, and FreeBSD? at the moment. In addition to the computers, a switch with enough ports to connect to all of the client machines is required. The best machine should be used as the server. This server requires two network devices: one for incoming traffic (The data to be captured) and the other to forward captured packets to the switch with the clients. Both of these network devices should be capable of keeping up with the data to be captured. A third network device is useful if remote connections are to be made to the server so as not to interfere with the capture.

II. Hardware Setup

Set up the server and all client machines. Connect the line containing data to be captured to a fast port on the server. Connect another line from the other fast network port on the server to the uplink port of the switch. Connect each client computer to a port on the switch. Set up the network appropriately such that the server has an IP address on the outbound (connected to the switch) port and such that the clients each have IPs. This way, a user on the server can control the clients using SSH or Telnet, so that the clients need not have keyboards or monitors.

III. Software Setup

Extract the ethermux tar file to a folder on the server and on each of the clients. In the base ethermux directory, issue the "make" command to build Cheap Logger for your system. If there are binary and object files within ethermux and its subdirectories, you may need to "make clean" before building the package. Documentation can be generated by typing "make doc", which will create HTML and Latex doucmentation for Ethermux in the "docs" folder. The server will be using the "ethermux" program located in the "server" directory, while the client computers will be using the "logger" program located in the "client" directory.

IV. Starting the System

Run the "logger" program on each of the client systems by running "client/logger". When the client loggers start, they will show the names of the dump files they have opened, as well as how full the packet buffer is. Command line options have not yet been finalized, so if changes must be made to the way the logger runs, do so by altering the #define statements in logger.c before compiling. Once the logger is running it will periodically try to connect to the ethermux program on the server. When it connects, it will display a message, and then start listening for packets from the server. Once all the clients are running the logger program, Ethermux must be started on the server. The basic command for this is "/server/ethermux -i InboundNetDevName? -o OutboundNetDevName? ". As an example, if your server is running Linux and is set up to listen on eth0 and forward packets to a switch on eth1, you would run "/server/ethermux -i eth0 -o eth1". You will most likely need to run the server as the root or administrator user because Ethermux uses libpcap for direct access to the network devices in promiscuous mode. When ethermux is running, it will accept all the client computers that have logger running on them. As soon as one client is connected, Ethermux begins capturing data to it.

V. Ending a Capture

The best way to end a capture is to kill the server first. Find the Quit option in the menus, or use Ctrl-C to safely shut Ethermux down. Once Ethermux has stopped, quit each client individually by pressing Ctrl-C on each system.

VI. The Data

The capture files reside on the client systems in the working directory from which logger was called. On each client, they are named "log.0001", "log.0002", and so forth. As of this early version of the logger, the last two or three log files may not contain any packets, as they were created preemptively to reduce load on the buffer during file changes. Empty logs are 24 bytes in size, Larger logs contain packets. The headers written for each packet by libpcap contain the timestamps at which they were received on the clients. In order to fix these timestamps to contain the times they were captured on the server, one must run the post processor. For information on how to do this, please see Dan Dugovic's documentation.