Professional Rootkits (Programmer to Programmer)

IDA, shown in Figure A-5, is by far the most complex tool detailed in this appendix. If you’ve read this book, you’ve already used IDA to peek at a few PE formatted files, but IDA does a lot more than just convert files into assembly language. Because of the extensive feature set and the capability to disassemble files for various processors, augment functionality with plug-ins, customize configuration files, comment and rename disassembled output, and much more, IDA can take quite some time to master. Fortunately, the default IDA configuration enables users to simply drag and drop a file onto the IDA shortcut (or the idaw.exe file) to get started.

Figure A-5

To begin, unzip the IDA archive into a newly created directory. There is no installation process, so you can use IDA immediately after unzipping the archive. If you create a shortcut for the IDA application, you should know that the working directory you select (the “Start in:” entry for the shortcut) will receive database files for every file you open and save with IDA. These database files, not the original files, are used the next time you select the original file. Therefore, if you move IDA around, or change the working directory, you can lose the modifications made to previously loaded files.

There are two main configuration files for IDA: IDA.CFG and IDATUI.CFG. Looking at these files with a text editor can give you an idea of the many options available in IDA. Modifying either of these files can quickly demonstrate the complexity of the IDA disassembler, so remember to make backups, or be prepared to reinstall after making changes to these files.

There are many IDA plug-ins available from the Internet. Some of these plug-ins can make using IDA much easier. Some can make IDA much more powerful. Some come with detailed instructions detailing how to configure and integrate the plug-in. Some come with tutorials that show how to use the plug-in. However, remember that IDA is a hacker’s tool, so expect some plug-ins to come with rootkits!

Once a file is selected, IDA will ask a few questions before loading. In most circumstances, the default values will enable IDA to properly process the file. If you know the selected file is not PE formatted (DOS or binary), you will need to select the proper format. Otherwise, select OK when presented with the Load File dialog. You can be presented with additional dialogs depending upon the file selected and the IDA configuration, but the default is usually the proper answer to all dialogs presented when a file is first selected.

Once a file is loaded, you can view many “windows” detailing the individual sections of the selected file. If you’ve already seen IDA, you know the term “windows” is used loosely here. In actuality, IDA windows are constructed using special ASCII characters, not graphics, so working with IDA windows can take a little time to get used to. After you are accustomed to this text-based convention, however, navigating the many windows of IDA can be as easy as navigating graphic windows. Just keep in mind that IDA windows are bordered by double lines, the upper-left corner contains a square that will close the window when clicked, the lower-right contains a square that can be dragged to size the window, and scroll bars are located at the right and bottom of every “window.”

The main window for most reverse engineering tasks is the Disassembly window. This is the only window initially opened in a default configuration. If you resize this window immediately after loading a file, you will see the status area immediately beneath the Disassembly window. The status area contains the log entries accumulated since the file was opened, and it is always available as the bottommost layer of the IDA user interface.

In addition to the Disassembly window, the following windows can be opened from the View menu option: Function, Names, Signatures, Segments, Segment Registers, Selectors, Cross references, Structures, Enumerations, and Problems. Some of these windows will be empty and some will not be available depending upon the type and content of the file selected.

Several options in IDA make reverse engineering much easier. The first is function jumping. If you already know the name of the function you wish to investigate, you can select Navigate Jump to, and then double-click a function to see the disassembled code for that function. While in the Jump to dialog, you can use the scroll bar or just begin typing the name of the function. You can also jump to a function by double-clicking any reference to the function in the disassembled output.

Another handy IDA feature is the capability to view the machine code for a particular function. This is extremely useful when signature strings are required to find non-exported functions within Dynamic Link Libraries. To modify the Disassembly window to show both assembly and machine code, select the Options Text representation menu option and enter a positive number (I recommend 8) into the Number of Op Code Bytes field.

For more information on using IDA, I recommend a Google search that includes the keywords ida, disassembler, and tutorial.