Critical Incident Management

In many descriptions you will see the words "Critical Incident Response Team" associated with critical incidents. Many incident response efforts are unsuccessful, not for lack of planning, but because many mistakes were made in creating a team that was not staffed with knowledgeable, dedicated employees. Many organizations use checklist methods of emergency response because of legal or policy mandates where senior managers think their systems' security is guaranteed because they mark a box. Feeling they have met all legal and policy requirements, they are lulled into a false sense of security.

Security controls have the purpose of making unauthorized entry so unattractive and difficult, they compel attackers to go elsewhere. The only truly effective security systems are those that render important systems inoperable. Of course, that condition is ridiculous. Systems security before, during, and after a critical incident exists as part of the whole picture of good business practice. It ensures uptime, efficiency providing critical systems needed for daily business operations. The purpose of security is to preserve what belongs to the organization from being stolen, deleted, or modified. So, what happens when an attacker, inside or outside the organization, causes a critical incident?

Most organizations have long understood the importance of having fire suppression equipment installed in data-centers, emergency exits, and employee training for emergencies. These same organizations have extensive information security measures with firewalls, DMZs, VPNs, and physical security. Safeguards, like these, have the purpose of maintaining the organization's property and reputation in the community.

Regrettably, critical incident response and management are often neglected until a catastrophe actually strikes and the organization finds itself scrambling to recover.

CIRT

CIRTs should be composed of team members with specific roles supported by specialized training and experience. The CIRT must have a function-point or coordinator where all reports of critical incidents are made. The function-point is usually an individual senior employee or member of a business unit having significant managerial and business experience. She possesses a clear understanding of the organization's goals and objectives, and probably participates in the drafting of the business' operational plans sometime in her career.

It is not expected this person would have a complete knowledge of the organization's mission, goals, policies, and procedures, but it is important that she have sufficient knowledge. For the function-point person to deliver services, she must be available 24 hours, holidays, and weekends. Contact may be accomplished through telephone or other expedient means.

Under practical circumstances, it is immaterial whether the organization decides to use its own in-house talent or delegates the responsibility to outside consultants. The first contact is the employee who receives information relating to the critical incident and makes several important decisions relating to it:

• Does an actual critical incident exist?

• Where is the critical incident occurring?

• What is the extent of the damage?

• Has the damage been contained or is it continuing?

• Do I need to triage the damage at this moment?

• What resources do I need to deploy at this moment?

• Do I have the resources to address this crisis at this time?

• Do I have sufficient information to deliver a meaningful report to senior managers?

• When should I notify law enforcement authorities?

Using Outside Consultants

One of the greatest advantages in using outside consultants (commercial CIRTs) is that of overall reduced cost. This is particularly true in smaller organizations where their operational demands are less than larger organizations. In many cases, contract consultants specializing in critical incident response deal with a wide variety of matters resulting in a high degree of expertise. Additionally, many of their team members have specialties such as UNIX, Windows, or specific programming languages usually not available to employees of smaller businesses.

These are the advantages of commercial CIRTs:

• Most commercial firms have the ability to respond in a matter of hours depending on travel times.

• They provide 24-hour support and are in constant contact.

• They can offer full-service response-posture, as their services usually include forensic duplication and examination, litigation support, expert testimony, technical support, policy formulation, and legal expertise.

• Commercial CIRTs can provide mock-incident response training. Participants address imaginary, but logical, scenarios and interact with personnel, data, and facilities.

• Keeping abreast of current attack-trends. Commercial CIRTs are able to track attacker trends and tailor their response-posture to their clients. By assigning technically trained account executives to clients, they anticipate malicious behavior and are prepared to marshal resources accordingly.

Commercial CIRTs vary greatly in their abilities. Senior managers should do their homework before signing contracts for service.

• Be certain to ask for references from several recent customers and do not hesitate to ask for individual employee's qualifications and experience levels.

• Contact their references. Ask the most important question, "would you hire them again?"

• Determine their reputation in the business community by contact entities such as the Better Business Bureau to ascertain if complaints have been filed and are unresolved.

• Depending on circumstances, ask for financial references.

• Determine if they have bonding in the event of future legal action.

Using In-House Talent

The primary reason for initiating and developing an in-house CIRT is the ability to address emergencies observing the organization's policies and procedures. Staffed with employees, CIRT capability can be directed to address emergencies meeting cultural and internal needs. Because critical incidents often involve sensitive or political matters, in-house talents are more likely to address them in a fashion most advantageous to the organization.

In many cases, internal CIRTs are funded through the corporate offices or on a charge-back basis to the individual business units. Some CIRTs are funded through corporate headquarters paying salaries and other recurring expenses while the individual business units pay for the on-site expenses such as travel, lodging, or other expenses.

Here are a few advantages of the internal CIRTs:

• Direct support. Internal CIRTs will provide emergency response to affected business units with greater specific business-practice knowledge than commercial CIRTs. Generally, they have greater sensitivity to corporate culture than equivalent outside firms.

• Risk management, policy, and audit support. Although these functions are usually addressed by an organization's business units having an internal CIRT can provide invaluable input to heightened awareness and effectiveness. After all, the CIRT has a high vantage point from which to gauge their interaction and deliver this experience to risk managers, policy writers, and auditors on a continuing basis.

• Emergency drill participation. An internal CIRT can participate in emergency exercises testing the full range of recovery, resumption, and critical incident response capabilities. An emergency exercise consisting of an unannounced test can measure the effectiveness of personnel, equipment, and procedures. Postmortem critiques, conducted among employees, are generally more productive and sensitive than sessions involving outsiders.

Ad Hoc CIRTs

This is a concept that has gained a lot of favor in the past few years for smaller businesses. Ad hoc CIRTs are developed utilizing existing talent, and where deficiencies are identified training is vigorously sought. For the most part, ad hoc CIRTs are composed of specially trained employees that have regularly assigned duties and when emergencies strike, they form their response team. For this concept to avoid being stillborn, it must have fanatical senior management sponsorship.

Here are a few suggestions for getting an ad hoc version of CIRT off the ground:

• Identify key technical employees that are qualified to address critical incidents. Such experts would include the IT manager, senior systems administrator, senior engineers, legal advisor, risk manager, human resources unit, etc.

• Draft response policies and procedures for the CIRT to screen initial reports, criteria for activation, response activities, and post-incident critique.

• Obtain senior management agreements with a minimum number of hours of participation on an annual basis for CIRT members. Provide financial incentives to employees for CIRT participation.

• Provide specialized training to CIRT members. This should be training that is complementary with their skills.

• Seek to train toward professional certifications. This is one of those incentive areas where CIRT participants can receive certifications qualifying them for advancement.

CIRT Requirements and Roles

As in any plan, the best place to start is with your deliverables and requirements. Experienced planners actually begin at the end by asking, "What is it we need the CIRT to do?" The most basic requirement for an incident response team is providing support and direction in successfully resolving critical incidents with a minimum degree of business disruption.

Basically, CIRTs are support units intended to provide critical incident response support to the organization as a whole and to the affected business unit specifically.

In this tasking, CIRTs usually serve in these potential roles:

• Direct hands-on emergency response where CIRT members are actively engaged in the containment and restoration of critical IT functions. The full-version of this activity is for the CIRT to assume complete response responsibility. Taking this posture potentially alienates employees already assigned to the affected business units. However, in the event of severe circumstances and if mandated by senior managers, this approach is efficient and effective.

• However, this role can suffer from a conflict of loyalties, as the CIRT is sometimes regarded as "big brother" when it appears on the scene and immediately takes control of the situation.

• Advisory/Shared role. In this role, the CIRT acts as a trusted advisor sharing response activities with the affected business unit. There is less conflict of loyalties in this role, meaning responsibilities are shared between entities.

Added CIRT Responsibilities

Because senior managers view full-time CIRTs as responding only when needed, sometimes they get the reputation of having little if anything to do unless they are responding to a crisis. Their perceived usefulness can be expanded by accepting added responsibilities:

• Acting as a problem screening unit. In this capacity, the CIRT acts as a unit where software patches, tools, and updated software versions are tried and tested before being applied. The practical side of this task rests in the CIRT being able to patch a corrupted system and know the patch they are applying has been tested. There is confidence this patch will not conflict with existing systems and is free from malicious code. Additionally, the CIRT acts like a clearinghouse for recurring or particularly troublesome system problems. They work closely with help desk coordinators and system administrators where any indications of critical incidents are reported and a determination is made if the CIRT should be activated either as an entire unit or in part.

• Coordinate inside emergency efforts and establish liaison with outside agencies. The CIRT coordinates the emergency response efforts of all organizational units in the event of a crisis and works to actively facilitate their drill and actual crises. The CIRT is tasked with the development of liaison with law enforcement and regulatory and legal entities. It actively seeks to participate in such entities as NIPC (National Infrastructure Protection Center), Infragard, HTCIA (High Tech Crime Investigators Association), ISACA (Information Security Audit and Control Association), ISSA (Information Systems Security Association), and the ACFE (Association of Certified Fraud Examiners).

• Provide training inside the organization and to outside entities. CIRT members should be in a very good position to deliver specialized training and increased awareness as one of their proactive jobs. Consider having the CIRT author technical articles in professional periodicals thereby benefiting them by having to do the research and delivering information to other professionals. Through this means, team members learn about developments and emerging trends while potentially providing a valuable service to their constituents and their organization.

CIRT Funding

Funding CIRTs, as are most business matters, is merely a matter of funding. Sometimes developing resources is more a matter of convincing bean counters of their value than anything else.

Here are a few basics to consider when developing your CIRT:

• CIRT as part of the IT function. Locating a CIRT as part of the organization's IT function can greatly facilitate productive interaction between the lessons learned as a result of responding to emergencies and improving development processes. Placing the CIRT as part of the IT function creates avenues of communication between responders and systems staff.

• Business units may benefit by having the CIRT as part of their operation. For example, the systems development unit could greatly benefit by having the knowledge and skills of the CIRT integrated as part of their operation. Having the CIRT as part of the IT audit unit could provide increased granularity and direction in audit programs.

• Corporate headquarters may wish to fund the cost of the CIRT's activities charged as an overhead cost to each of its business units. In this fashion, the cost of having the CIRT is spread to all affected business units, saving each unit from having to make preparations and fund critical response programs. In this fashion, there is an avoidance of duplicating response efforts between headquarters and the individual business units saving time and money. By adopting the "big picture" view, it allows the CIRT to respond to emergencies on the corporate level where trouble spots can be more readily recognized and addressed.

Who Does the CIRT Support?

The quick answer to this question is everybody. However, for a CIRT to adequately function, it must understand the people it serves mission and goals. For CIRT managers, it is suggested they track units calling for their services so they may gear their response accordingly. It is likely the same business units are requesting services time and again; consequently, it is important for CIRT to service their requests as if they were favored clients. For example, if the business units primarily supported by the CIRT consisted of systems users rather than findings produced by IT audit reports, CIRT's response would be less technical than the response delivered to the auditor's findings. Responding to the auditors would probably require more forensic skills than responding to worms and viruses encountered by users.

CIRT Communications

CIRT members should be mindful that their clients are the business units they service. Misplaced, flippant, or capricious remarks return poor dividends. Communications between the CIRT and the units it supports is not just something that is casually performed; it must be a matter of deliberation and coordinated efforts.

CIRTs should have specific communications goals when measuring their success:

• Timeliness. CIRTs must deliver information in a timely fashion. The means by which the information is transmitted may be e-mail, telephone calls, faxes, voice mail, company Web sites, memos, conferences and workshops, working groups, seminars, and bulletin boards. Basically, employees served by the CIRT should have information as soon as it is discovered. For example, the CIRT becomes aware that the BUGBEAR.exe virus is in the wild. The most efficient way to deliver a message warning about the proliferation and damage this virus can cause is by sending a voice mail message to each employee warning that e-mail attachments should not be opened. Sending an e-mail to each employee may result in the information arriving too late, as the employee may be checking e-mail by opening an attachment before getting to the CIRT warning. Timely and credible warnings will go a long way to developing the CIRT's position and credibility in the organization.

• Relevant communications are a must for a CIRT. If the units supported by them are primarily Windows platforms, it does little good to deliver information about UNIX and OS/2. Communications should be crafted so they are meaningful to their recipients.

• Digestibility. The intended audience must understand CIRT communications. For example, if the CIRT primarily serves workstation users, the CIRT should not craft exhaustive communications dealing with the technical aspects of UNIX server configurations. Granted, there might be readers who enjoy the finer aspects of server configurations, but the broader appeal will be to the majority of the users. Reserve specialized information to specialized employees.

  CIRTs should be mindful that there are many levels of employees that are going to read their material, including managers. Including a brief executive summary at the beginning of the communication is appreciated. Depending on the audience, it may serve to have two or even three versions of a communication to be disseminated. One version would be delivered to the general user population, one version to be sent to the managers, and another version intended for the technical staff.

• Accuracy of communications. Few actions will work to destroy the credibility of any business unit faster than to disseminate incorrect information. Get the facts, and get them straight before transmitting information to anyone. Every phrase and every term should be carefully scrutinized for accuracy before going out. CIRT managers must read the communication for technical accuracy and understanding. Of course, CIRT communications should be professional and courteous. This is not the place for colorless humor or sarcasm. Part of communication's accuracy is the assurance the intended audience receives them.

CIRTs should develop out-of-band communications. This means that CIRTs, their constituency, and management should know when and how to use OOBC. OOBC efforts require advance arrangements and coordination within a response team. CIRTs should analyze the organization's current communication structure and devise private alternate channels. OOBC may include private cellular telephones, text pagers, wireless equipment such as PDAs, out-of-business-area telephone communications, registered mail, encrypted e-mail, etc. CIRTs must ensure that each OOBC system is periodically tested and achieves acceptable levels of security.

Developing Critical Incident Cost Analyses

CIRTs should develop a means by which they can measure the cost of addressing critical incidents. The reason for this procedure is fairly simple, if their organization is going to pursue legal actions to recover damages, or criminal sentencing is directly tied to the amount of damage done, a monetary amount is necessary.

In recent years, almost everyone with an e-mail address has heard of or experienced the damage caused by the Melissa, SirCam, CodeRed, Nimda, Slammer, and Klez viruses.

Ask users and administrators how much "damage" they suffered as a result of these and other pieces of malicious software and you will hear, "Well, not much, I guess." From their perspective, how much does it cost to reformat hard drives and reinstall operating systems, applications, and data?

Adding up the time lost to handling critical incidents across a large organization and you are talking sizeable money amounts. If corporate executives, government administrators, or university regents were asked how much money is being lost to such incidents, they would likely answer they do not have any idea and they do not have any mechanism to collect such data.

The matter is simple if you know a few facts about responding CIRTs and affected systems. Such information can be obtained by answering a few questions:

• Who responded to the incident?

• What equipment and programs were used?

• What post-incident analysis was performed pursuant to the incident?

• How many hours did each of them spend in that response? What are their salary levels?

• How many employees were prevented from working because of the emergency?

• How many hours of employee downtime were attributable to the emergency?

• Calculating an estimated amount of Internet business, what were the anticipated business losses attributable to not being able to conduct e-Business?

• What are the calculated overhead costs (insurance, sick leave, etc.) for each employee not able to work due to the critical incident?

The largest obstacle in obtaining such cost estimates is motivating employees to keep track of their billable time. CIRTs and systems administrators are usually anxious to return the system to productivity and usually do not keep careful notes of what they did and when they did it. However, if a civil action is filed or criminal charges leveled, the best means of ensuring accurate testimony is the employees' recollection supported by their notes.

Exhibit 14 is a table reflecting the costs related to a small incident.

Exhibit 14