%\documentclass[10pt,twoside,twocolumn]{article}
\documentclass[10pt,twoside]{article}
\usepackage[bf,small,nooneline]{caption}
\usepackage[letterpaper,hmargin=1in,vmargin=1in]{geometry}
\usepackage{paralist} % comapctitem, compactdesc, compactenum
\usepackage{titlesec}
\usepackage{titletoc}
\usepackage{times}
\usepackage{hyperref}
%\usepackage{glossaries}
%\usepackage[xindy]{glossaries}
\usepackage[toc]{glossaries}
\usepackage{graphicx}
\graphicspath{{./graphics/}}
\usepackage{xspace}
\usepackage{verbatim}
\hyphenation{Sub-Bytes Shift-Rows Mix-Col-umns Add-Round-Key}

\newcommand{\bulk}{\emph{bulk\_extractor}\xspace}
\newcommand{\bev}{\emph{BEViewer}\xspace}
\newcommand{\button}[1]{(\raisebox{-1.20mm}{\includegraphics[scale=0.90]{./graphics/#1}})}

\newglossaryentry{BEViewer}
{name={\emph{BEViewer}},description={The Bulk Extractor Viewer User Interface application}}

\newglossaryentry{BulkExtractor}
{name={\emph{bulk\_extractor}},description={The utility for extracting Features from Image media.
\bulk also reads embedded Paths for \bev}}

\newglossaryentry{Feature}
{name={Feature}, description={A sequence of bytes that are specifically searched for
and found in an Image during a \bulk scan}}

\newglossaryentry{FeatureFile}
{name={Feature File}, description={A File containing Features, created during a \bulk scan}}

\newglossaryentry{HistogramFile}
{name={Histogram File}, description={A File containing a histogram of Features
produced from its associated Feature File, created during a \bulk scan}}

\newglossaryentry{Histogram}
{name={Histogram}, description={The number of occurrences (frequency)
of various Features in a Feature File}}

\newglossaryentry{ReferencedFeatureFile}
{name={Referenced Feature File}, description={A Feature File that is referenced to
by its associated Histogram file.
For example the referenced Feature File for file \texttt{email\_histogram.txt}
is \texttt{email.txt}.
\bev displays the Referenced Feature File when a Histogram file is selected in the Reports View}}

\newglossaryentry{ReferencedFeature}
{name={Referenced Feature}, description={The Features contained within a Referenced Feature File}}

\newglossaryentry{Bookmark}
{name={Bookmark}, description={To add a Feature to the set of Bookmarked Features
so that it can be saved in a Case or placed in a Report}}

\newglossaryentry{BookmarkedFeature}
{name={Bookmarked Feature}, description={A Feature that has been Bookmarked.
Bookmarked Features may be exported as Case Evidence}}

\newglossaryentry{Report}
{name={Report}, description={The Feature Files and associated information
generated by running \bulk.
\bev opens Report directories created by \bulk
so that Features in their Feature Files may be Navigated to}}

\newglossaryentry{Case}
{name={Case}, description={The collection of Reports and Bookmarks
associated with a Forensics Investigation performed by an Examiner using \bev.
Cases may be opened and saved}}

\newglossaryentry{Navigate}
{name={Navigate}, description={To show a Feature, its attributes, and a page of its Image View
in the Navigation Area}}

\newglossaryentry{NavigationHistory}
{name={Navigation History}, description={The collection of Features that have been Navigated to
during the \bev session.
The list of Features is lost when \bev closes.
To make Features persistent, Bookmark them}}

\newglossaryentry{Path}
{name={Path}, description={The location of a Feature in an Image.
Direct paths are represented by an offset into the Image.
Compressed paths are represented by syntax indicating how the Feature resides in compressed form,
and includes the offset into the decompressed space}}

\newglossaryentry{Offset}
{name={Offset}, description={The location into an Image
where a Feature or compressed region resides}}

\newglossaryentry{ImageFile}
{name={Image File}, description={A file containing a Forensics Image from digital media.
Image Files may be contained in any of several Forensics formats such as \texttt{.E01},
\texttt{.001}, and {.aff}}}

\newglossaryentry{Image}
{name={Image}, description={The actual Image media, regardless of the file format that contains it}}

\newglossaryentry{AddressBase}
{name={Address Base}, description={The numeric base used for displaying Offset values,
specificallly Decimal or Hexadecimal}}

\newglossaryentry{Highlight}
{name={Highlight}, description={To color the background around Feature text and Image text
to make it more visible.
Feature text or typed text may be highlighted,
depending on the Highlight Source selected by the user}}

\newglossaryentry{HighlightSource}
{name={Highlight Source}, description={The source to use for highlighting text,
either the Navigated Feature or else typed text}}

\newglossaryentry{Log}
{name={Log}, description={A record of user actions and internal operations maintained by \bev
useful for diagnostics in the event of an error}}

\newglossaryentry{Filter}
{name={Filter}, description={Text used for controlling the list of Features shown
in the Features View.
Only Features matching this text are displayed.
Sensitivity to capitalization may also be set}}

\newglossaryentry{SystemClipboard}
{name={System Clipboard}, description={A System scratch pad used for copying
Features in the Features Area and Image bytes in the Navigation Area
from \bev into other applications running on the system}}

\newglossaryentry{ImageReader}
{name={Image Reader}, description={The internal reader that \bev uses for reading Image bytes
from Image Files.
Although \bev selects the appropriate Image Reader to use,
a specific Image Reader may be set for testing purposes}}

\makeglossaries

\begin{document}

%\titlespacing{\section}{0pt}{0pt}{0pt}
%\titlespacing{\subsection}{0pt}{0pt}{0pt}
%\titlespacing{\subsubsection}{0pt}{0pt}{0pt}
%\date{}

\title{Bulk Extractor Viewer (\bev) User Manual}
\author{Bruce Allen \footnote{\href{mailto:bdallen@nps.edu}{bdallen@nps.edu}}}
\maketitle

%\begin{abstract}
%  The Bulk Extractor Viewer (\bev)
%  is a User Interface for browsing Feature data extracted
%  using the \bulk extraction tool.
%  In addition to browsing Features,
%  \bev provides facilities for viewing multiple Images,
%  managing Feature sets as Cases,
%  and exporting Bookmarked Features.
%\end{abstract}

%\newpage
\cleardoublepage
\setcounter{tocdepth}{2}
\tableofcontents
%\newpage
\cleardoublepage

\section{Introduction}
\bulk and \bev together provide rapid Forensics triage of digital media.
\bulk rapidly identifies \glspl{Feature} in a Forensics \gls{Image}
and produces a \gls{Histogram} identifying the frequency of Features.
\bev provides a User Interface for browsing through Features identified by \bulk
and for preparing \glspl{Case} and managing \glspl{Report}.

\subsection{\bulk Capabilities}
\bulk scans Forensic \glspl{Image} to produce \glspl{Feature}.
\bulk also provides facilities for reading embedded media
such as content within an embedded \texttt{.zip} file.

\bulk provides the following capabilities:
\begin{compactitem}

\item \bulk Captures \glspl{Feature} of various types:
% this compactitem list is copied from bulk_extractor/bulk_extractor.tex
\begin{compactitem}
\item Email addresses
\item Credit card numbers, including track 2 information
\item Search terms (extracted from URLs)
\item Phone numbers
\item GPS coordinates
\item EXIF (Exchangeable Image File Format) information from JPEG
  images
\item A list of all words present on the disk, for use in
  password cracking
\end{compactitem}

\item \bulk runs on Windows, Linux and Macintosh-based systems.
\item \bulk operates on raw disk images, split-raw volumes, EnCase E01 files, and AFF files.
\item \bulk additionally extracts features
from compressed data such as ZIP and windows hibernation files.
\item \bulk employs Thread Pool optimization
which fully utilizes each processor on the system to maximize processing performance.
\item \bulk can extract regular expressions provided as user input.
\item \bulk runs plug-ins for custom Image processing.
\item \bulk recognizes context-sensitive stop-lists.
\end{compactitem}

\subsection{\bev Capabilities}
The \gls{BEViewer} User Interface allows the user
to browse through \glspl{Feature} in \glspl{FeatureFile} created
using the \gls{BulkExtractor} tool \cite{garfinkel:bulk-extractor}.
\bev also provides interfaces for managing \glspl{Case}, generating \glspl{Report},
and running \bulk scans.

\bev provides the following capabilities:
\begin{compactitem}
\item Browse \glspl{Feature}:
\begin{compactitem}
\item View Features in \glspl{FeatureFile} created by \bulk.
\item Scroll the \gls{Image} view to see Features in their context.
\item View Features from multiple Images.
%\item Open multiple \glspl{Report} from multiple \glspl{Image} simultaneously.
\item \gls{Navigate} between Features using a History list.
\item \gls{Bookmark} features for including them in Cases and Reports.
\item \gls{Highlight} Feature content for easy visual identification.
\item Copy Feature and Image content to the \gls{SystemClipboard}.
\item Filter Features to find specific text.
\end{compactitem}
\item Manage \glspl{Case}:
\begin{compactitem}
\item Bookmark Features to include them in a Case.
\item Save and open Case settings as Case files.
\item Transfer Case files to other Examiners or to another computer.
\end{compactitem}
\item Generate \glspl{Report}:
\begin{compactitem}
\item Bookmark Features to be placed in Reports.
\item Manage the Bookmarked Features list.
\item Export Bookmarked Features to a file for reporting.
\end{compactitem}
\item Run \gls{BulkExtractor}:
\begin{compactitem}
\item Launch the \bulk extraction tool.
\item Monitor the progress of \bulk as it runs.
\end{compactitem}
\end{compactitem}

\section{Downloading \bev}
Please download \bev from github at \url{https://github.com/simsong/bulk_extractor/downloads}.

\begin{compactitem}
\item On Windows, please download and run the Windows installer.
This installation includes \bulk.
\item On Macintosh or Linux, please run the \bev \texttt{.jar} file from the command prompt.
\end{compactitem}

\section {Running \bulk from the Command Line}
Although the \bev runs \bulk directly,
it may be useful to invoke \bulk from the command line.
\bulk usage parameters follows:

\subsection*{Basic Usage}
Basic usage is: \texttt{bulk\_extractor [options] imagefile}.

\subsection*{Required parameters}
\begin{description}
\item [\texttt{<imagefile> | -R <filedir>}]
The input source to extract \glspl{Feature} from.
This source may be either an \gls{Image} file
or a directory containing files.
\item [\texttt{-o outdir}] The new output directory that \bulk will create.
This directory must not exist unless \bulk is performing a recovery after a crash
and the directory in progress is valid.
\item[\texttt{-b banner.txt}] Adds \texttt{banner.txt} contents to the top and bottom of specific
output files generated by \bulk.
\item[\texttt{-r alert\_list.txt}] Defines the alert list of features.
This list can contan regular expressions.
\item[\texttt{-w stop\_list.txt}] Defines the stop list (white list) of features.
This list can contan regular expressions.
\item[\texttt{-F <rfile>}] Read a list of regular expressions from <rfile> to find.
\item[\texttt{-f <regex>}] find occurances of <regex>; may be repeated.
                  results go into find.txt.
\item[\texttt{-q nn}] Quiet Rate; only print every nn status reports. Default 0;
                 -1 for no status at all.
\end{description}

\subsection*{Tuning parameters}
\begin{description}
\item[\texttt{-C NN}] Specifies the size of the context window, default 16.
\item[\texttt{-G NN}] specifies the page size (default 4194304).
\item[\texttt{-g NN}] specifies the margin (default 4194304).
\item[\texttt{-W n1:n2}] specifies the minimum and maximum word size.
Default values are 6 and 14.
\item[\texttt{-B nn}] Specify the blocksize for bulk data analysis (default 512).
\item[\texttt{-j NN}] specifies the number of threads to run, default is \#CPUs.
\end{description}

\subsection*{Path Processing Mode}
\begin{description}
\item[\texttt{-p [[<path>/[r | h]] | - | -http]}] Prints the Image content at the requested path.
For \texttt{\texttt{-p <path>/[r | h]}}, the path is printed
with format \texttt{r} for raw byte output or \texttt{h} for a hexadecimal dump.
For \texttt{-p -}, \bulk enters interactive mode, and takes ? as input to produce 4K output
as a hexadecimal dump.
For \texttt{-p -http}, \bulk enters interactive mode for servicing HTTP v.1.1 GET requests
to produce binary output.
\end{description}

\subsection*{Parallelizing}
\begin{description}
\item[\texttt{-Y <o1>}] Start processing at o1 (o1 may be 1, 1K, 1M or 1G).
\item[\texttt{-Y <o1>-<o2>}] process o1-o2.
\item[\texttt{-A <off>}] Add <off> to all reported feature offsets.
\end{description}

\subsection*{Debugging}
\begin{description}
\item[\texttt{-h}] print the Usage help message.
\item[\texttt{-H}] print detailed info on the scanners.
\item[\texttt{-V}] print version number.
\item[\texttt{-M nn}] Sets the maximum recursion depth for recursive scanning
within compressed Image regions, default 5.
This limit protects against compression bombs.
\item[\texttt{-z nn}] Start on page \texttt{nn}.
\item[\texttt{-dN}] debug mode (see source code).
\item[\texttt{-Z nn}] zap (erase) output directory.
\end{description}

\subsection*{Control of Scanners}
\begin{description}
\item[\texttt{-P <dir>}] Specifies a plugin directory.
\item[\texttt{-E scanner}] Turn off all scanners except \texttt{scanner}.
Valid scanner names to be isolated are:
\item[\texttt{-m <max>}] maximum number of minutes to wait for memory starvation
                  default is 60.
\item[\texttt{-s name=value}] sets a bulk extractor option name to be value.
\item[\texttt{-e <scanner>}] Enable scanner <scanner>.
\item[\texttt{-x <scanner>}] Disable scanner <scanner>.
\end{description}


\section{Running \bev}
Please see the \bev User documentation for information on running \bev.
Currently, this is available as a .pdf document on github.

\section{Technical Support}
A Bulk Extractor Users Group is available at \url{bulk\_extractor-users@googlegroups.com}.
This Users Group exists to promote discussions about the use of \bulk and \bev
and to identify future functionality that may benefit Forensics examiners.

For technical support, please email the Bulk Extractor User Group
at \href{mailto:bulk\_extractor-users@googlegroups.com}{bulk\_extractor-users@googlegroups.com}
or contact the Developer, Bruce Allen, at \href{mailto:bdallen@nps.edu}{bdallen@nps.edu}.

Please post new Feature recommendations, change requests,
or usability comments to the Bulk Extractor User Group or to the Developer.
Please also browse the list of proposed Future Work
posted under \url{http://domex.nps.edu/deep/Bulk\_Extractor.html}.

\section{Diagnostics}
\subsection{Crash Diagnosis}
Diagnosing and Overcoming \bulk crashes

All forensic programs occasionally crash on some input files, and \bulk is no different. Given some input files, \bulk will crash. These crashes are usually the result of input data that is corrupted or damaged in some way. Unfortunately we can’t go back to the bad guys and ask them to give us clean data, so forensic software needs to be tolerant of any kind of input. Although it is possible to write such software, it can be exceedingly difficult to cover every possible data corruption case.

The good news is that \bulk has a variety of command-line switches that you can use to get past most crashes.

\bulk is organized as a framework treats your disk image as a series of 16 Megabyte pages. Each page is read one-by-one and processed by one or more scanners. Most scanners look for a single kind of data and store it in the feature\_files. For example, the email scanner looks for email addresses and domain names and stores them in the email.txt and domain.txt files.

Some \bulk scanners are recursive. When these scanners find certain data in the page they will transform it into a new page and then recursively re-analyze it will all of the scanners. The GZIP and ZIP scanners, for example, will recognize compressed data, decompress it, and then re-analyze it. The pages analyzed by the recursive scanners can be of any size, from a single byte to hundreds of megabytes.

You can control \bulk’s behavior by turning on or off individual scanners. To find out which scanners are enabled by default, run \bulk with the “-h” option.

STEP ONE: VERIFY IF A SCANNER IS AT FAULT

The first thing to do after a crash is to verify that the crash is with a scanner and not with the \bulk framework. You can do this by running \bulk with a single scanner that doesn’t look for anything. The easiest way to do this is with the bulk scanner.

The “-E” flag turns off all scanners except the scanner specified after the flag. It can be followed by additional flags to turn on individual scanners.

So if you get a crash, the first thing to do is to verify that the framework is enact.

% \bulk -o out10 -E bulk disk_image.raw

(Where out10 is the output directory and disk\_image.raw is the disk image.)

STEP TWO: FIND THE SCANNER THAT’S MAKING YOU CRASH

The next step is to disable scanners one-by-one until you find the scanner that’s making \bulk crash. Typically this will be a single scanner. You can find out which scanners are enabled by default with the “-h” option.

For example, here is the bottom of a typical -h output:

Control of Scanners (feature recorders):
\begin{verbatim}
-E scanner – turn off all scanners except scanner
-x accts – disable scanner accts (ccn/c ccn_track2/c telephone/c )
-x base64 – disable scanner base64 ()
-x email – disable scanner email (domain/c email/c rfc822/c url/c )
-x exif – disable scanner exif (exif/c )
-x zip – disable scanner zip (zip )
-x gzip – disable scanner gzip ()
-x pdf – disable scanner pdf ()
-x hiber – disable scanner hiber ()
-e net – enable scanner net (ether ip tcp )
-e bulk – enable scanner bulk ()
-e find – enable scanner find (find/c )
-e wordlist – enable scanner wordlist (wordlist )
\end{verbatim}

In this example, you can disabled the accts, base64, email, exit, zip, gzip, pdf and hiber scanners. So you might want to try the following command lines to see the one that stops the crashing:

\begin{verbatim}
% bulk_extractor -o out-xaccts -x accts disk_image.raw
% bulk_extractor -o out-base64 -x base64 disk_image.raw
% bulk_extractor -o out-xemail -x email disk_image.raw
% bulk_extractor -o out-xexif -x exif disk_image.raw
% bulk_extractor -o out-xzip -x zip disk_image.raw
% bulk_extractor -o out-xgzip -x gzip disk_image.raw
% bulk_extractor -o out-xpdf -x pdf disk_image.raw
% bulk_extractor -o out-xhiber -x hiber disk_image.raw
\end{verbatim}

Notice that in each case the output directory has been named to indicate which scanner was disabled.

In most cases that we have seen, the crash results from a combination of a regular scanner and a recursive scanner. This means that you may be able to stop the crash by turning off one of two scanners—for example, either the email scanner or the gzip scanner. Some experimentation may be necessary to find the optimal settings.

If you wish to assist in the debugging of \bulk, it is useful to find out the smallest number of scanners that, when enabled, cause \bulk to crash with your particular disk image. You can turn off all of the scanners except the email scanner with the “-E email” command. You can then enable individual scanners by preceding the scanner name with the “-e” command. For example, to run just email and gzip, use this command:

% \bulk -o out10 -E email -e gzip disk_image.raw

STEP THREE: INVESTIGATE OTHER PARAMETERS

There are several other parameters that you can adjust to perhaps get past a crash.

3a – Enable Crash Protection will allow \bulk to recover from certain kinds of crashes. Unfortunately the recovery mode is inconsistent. For this reason crash protection is disabled by default.

To enable Crash Protection, run \bulk with the “-c” command.

3b – Set the maximum recursion depth.

By default, \bulk will recurse a maximum of five times. You may discover that you can prevent it from crashing on some media by limiting the number of recursions to 1 or 2. To do that use the “-M” command:

% \bulk -o out10 -M 2 disk_image.raw

STEP FOUR: CAPTURE A TRACE

\bulk is multi-threaded, which means that multiple threads of execution run at the same time within the program. In standard use \bulk creates one thread to read all of the pages from the disk. Between 1 and NN threads are then created to process the pages. Pages are not processed roughly in order, although different different pages take different amounts of time to process, so it is likely that some pages will start processing before other pages finish.

You can have \bulk print every time it starts and exits a scanner with the “-d1″ option. It’s not clear that this will actually help you debug the problem you encounter, but it might.

\subsection{Example of Functonal Testing}example process of functional testing from Simson, 7/3/12 5:32AM:

I'm currently doing some testing on \bulk to explore the impacts on performance by changing the page size from 16MiB to 1MiB and the margin size from 1MiB to 15MiB. I can do that by using the -g and -G flags and swapping the arguments. George Dinolt had started on this work but didn't follow through.

My initial testing with the centos virtual machine found that the 1/15 approach led to 3x the time to run. If it recovers 3x the features, then that's good --- it means we had features that are too big to be found with the 1MiB margin. But I think that the centos disk is a special case, in that I think that the disk image contains a large number of very big compressed objects.  So I'm doing some comparison tests on one of the Terry disks.

During the course of the experiment I saw this happen:

\begin{verbatim}
 8:22:23 Offset 6643MB (32.37%) Done in  0:22:58 at 08:45:21
 8:22:30 Offset 6710MB (32.70%) Done in  0:22:51 at 08:45:21
 8:22:34 Offset 6777MB (33.02%) Done in  0:22:39 at 08:45:13
 8:22:43 Offset 6845MB (33.35%) Done in  0:22:38 at 08:45:21
zero length feature at (|6528524288)
 8:22:53 Offset 6912MB (33.68%) Done in  0:22:39 at 08:45:32
 8:23:02 Offset 6979MB (34.00%) Done in  0:22:36 at 08:45:38
 8:23:16 Offset 7046MB (34.33%) Done in  0:22:43 at 08:45:59
 8:23:27 Offset 7113MB (34.66%) Done in  0:22:44 at 08:46:11
\end{verbatim}


The zero length feature is a problem. I'm not sure what is causing it, but it may be symptomatic of a deeper bug. So I'm going to re-run \bulk under the debugger and set a break point at the zero length feature printout. I'm then going to determine why it happened and fix the rule or whatever is the cause of the problem.

This is what I mean by testing. Detailed testing, looking for weird things that the program does, and then fixing them. 

At the same time, I am taking notes of the differences in the timing tests. I'm not keeping as many notes as I should and I'm running too many experiments without recording the results. But in this case I'll store the results in the TESTING.txt file in a manner that others can benefit from the results.

\subsection{Filing a Bug Report}
When filing a bug report, please indicate:
\begin{compactenum}
\item Are you using Python 2 or Python 3? Which version?
\item Which version of dfxml.py are you using?
\item Which  version of fiwalk are you using?
\item What generated he DFXML file?
\end{compactenum}
You're welcome to send reports to developers, rather than to the list.

\subsection{Diagnosing a Trap}
\begin{compactenum}
\item Compile \bulk with symbols on and optimization turns off.
Specifically, run \texttt{./configure}
but specify no optimization: \texttt{./configure --with-noopt}.
\item Then run the GNU debugger (gdb): \texttt{gdb bulk\_extractor}.
\item Then, from the gdb prompt: \texttt{run}.
\end{compactenum}


\cleardoublepage
\bibliography{beviewer_old}
\bibliographystyle{plain}

\cleardoublepage
\printglossaries

\begin{comment}
\section{Vocabulary}
\bev
\bulk
Feature
Feature File
Histogram File
Histogram
Referenced Feature File
Referenced Features
Bookmarked Features
Report
Case
Navigate
Navigation History
Path
Offset
Image File
Image
Address Base
Highlight
Highlight Source
Filter
Log
System Clipboard
Image Reader
\end{comment}

\end{document}

