Critical Incident Management

CIRT core membership should include the senior manager sponsor, IT security program manager, representatives from the legal counsel unit, public relations unit, human resources unit, and the CIRT manager. The CIRT manager should be someone who is a senior employee who has significant knowledge of the organization's operations as well as an employee capable of making sound business decisions.

The IT program manager is the head of the organization's IT security program and might double as the CIRT manager. In the case of a critical incident spanning regions or countries, one IT critical incident manager should be named for each office with all strategic efforts coordinated at the headquarters level. This representative will be responsible for tactical decisions, triage functions, and local resource deployment. It is the IT security program manager, with senior manager's approval, that is responsible for authorizing any release of information about the incident to the press. However, the program manager should not be the individual disclosing information to the press. A public relations unit employee should make contacts with the press. Delegating press responsibility relieves the program manager from having to evade sensitive questions or even having to lie to the press corps. Regardless, the public relations unit is going to be the place where the institutional knowledge and experience in this area is going to be found.

Legal Unit

Activating the CIRT requires an opinion from senior managers and specifically from the legal unit representative that is knowledgeable about the relevant laws dealing with the organization and its functions, intellectual property, information security, and privacy. In the case of CIRT deployment, it is the legal unit's responsibility to ensure that the CIRT does not violate laws and regulations while responding to a critical incident. Knowledgeable and experienced legal advise become particularly important when CIRTs are directed to follow attackers with the objectives of locating, identifying, assisting, apprehension, and prosecution. Legal representatives must be more than attorneys with general knowledge; they must possess a thorough understanding of information technology, business functions and civil, administrative and criminal matters. Through their participation on the CIRT core, they must initiate and develop relationships with law enforcement and regulatory authorities, professional support groups such as NIPC and Infragard. Often, this employee will serve as the primary contact for law enforcement.

Public Relations

Depending on the organization's size and funding, having a public relations unit representative is a decided advantage. This employee addresses all media requests for information and similarly handles authorized press releases. It is expected this employee will have developed relationships with media organizations as well as specific news agency representatives.

Human Resources Unit

A senior representative from the human resources unit must be part of the CIRT core. This person ensures that the CIRT team's response efforts do not violate employees' rights. Also, this person will make certain that appropriate disciplinary standards are applied should an employee be found to be the source of a critical incident. In the event an employee is an unwitting part of an attack, or if the employee is a victim, certain rights might be granted within the scope of their employment. The human resources unit representative is responsible for seeing that an employee's reasonable expectation of privacy is respected or knowing whether an employee is entitled to union representation in the event of an interview.

IT Investigative, Analysis, and Forensic Experts

These CIRT members ensure that the response is performed in a methodical and deliberate fashion, making certain all relevant evidence is properly collected, preserved, and introduced at legal proceedings. CIRTs require their members to participate in addressing crises on an as-needed basis. Key participants should consist of IT security officers, systems administrators, telecommunications equipment specialists, database managers, engineers/software developers, and of course, systems owners.

IT Security Officers

Most organizations have individuals assigned full- or part-time to ensuring the security of systems. Often this employee performs duties in support of auditors, making certain the IT units are in compliance with the organization's policies, procedures, and standards. This employee helps in addressing attacks by knowing how the system was installed and configured before the attack. She will also be the person who provides CIRT with access and interpretation of logs.

Systems Administrators

These employees are the "bread-and-butter" individuals responsible for the day-to-day operation of the system, including hardware, software, and employee interaction with the system. Systems administrators should have in-depth knowledge of the function of the system's hardware, operations, and configurations. Depending on the organization, its culture and function, the systems administrators can provide immeasurable assistance to the CIRT.

Telecommunications Specialists

These employees are the ones who are most knowledgeable about the integration of the various components of the telephone and network border systems, including installation, security, configuration, and operation. Systems administrators sometimes perform this function in smaller organizations. These employees have intimate knowledge of the interaction between the various hardware/software components, cabling, telephone lines, PBXs, terminal equipment, routers, firewalls, gateways, and protocols like X.25 or Frame Relay. They are usually responsible for developing relationships with communications carriers including the interaction between the organization and the carrier's equipment.

Database Managers

Most organizations dealing with substantial amounts of data will employ database managers and administrators. These are the employees who have the responsibility of maintaining the integrity of the database; assessing the impact of proposed changes; and in the event of an attack, determining the effects of deletions, modifications, or additions.

Engineers/Software Developers

These employees have knowledge of the system's platforms and applications and how they interact with the hardware. They are the employees that know if the system is running according to design specifications.

System Owners

It is imperative that the systems owners be part of the CIRT, as it is their responsibility to see that the system personnel, data, and facilities are functioning effectively and efficiently. Owners should know the emergency response/recovery plans and their execution. They will be fully aware of backup and restoration procedures as well as equipment redundancies. Ultimately, the owners are responsible to the other stake-holders and will have to answer questions regarding the attack, including its effect on critical assets.

CIRT Management Skills

Possessing well-developed management skills is the single-most desirable attribute the CIRT team leader can have. When a critical incident arrives, it is incumbent on the CIRT manager to ensure the team has the requisite skills, resources, training, experience, motivation, and attitude. Managing a CIRT is not really very different than managing any business unit that is populated by field-specific experts. CIRT managers do not need to have great technical proficiency, but on the other hand, they should have sufficient knowledge to make qualified decisions concerning team priorities and tactical deployments.

Technical Skills

Technical skills are absolutely essential in determining CIRT's efficiency and effectiveness. There is also a matter of the team's credibility. If the team does not earn a reputation for being able to handle emergencies, they will not be contacted for help and no one will listen to their warnings or advice. CIRT's technical skills should span relevant operating systems (UNIX, Linux, Windows, etc.); networking skills; programming languages such as C++, PERL, Java, XML, and HTML; and hardware equipment such as firewall appliances, routers, etc. Electrical engineering experience is a plus.

Staffing CIRTs with professionals that have skills in all relevant areas is extremely difficult and expensive. Such employees are going to command high salaries and are probably out of reach of most organizations. If this is not within the organization's budget, find individuals who have expertise in one or more areas and task them to work as a team. Teams, permanent and ad hoc, are composed of employees having key skills that mentor others in developing new skills. Foster a team culture of mutual dependence and spirit, it will pay dividends in the future.

Team Skills

These skills are vital in the CIRT's successful operation. Team skills are focused on:

• Having a common vision of the job to be done

• Division of responsibilities

• Ability of seeing the next item to be done without prompting

• Knowing when to tell and when to ask

• Knowing when the task exceeds an individual's skills resulting in getting help from another team member

Developing team skills is a direct result of management skills, so good managers tend to engender good team skills.

Communication Skills

Team members must be able to cooperate and communicate with coworkers as well as write and deliver effective formal presentations. If there are not employees that have technical writing skills, consider hiring technical writers to supplement team skills. Communications skills are so vital to CIRT's success, that if they are absent it is very possible that no amount of technical ability will compensate.

People Skills

In the event of a critical incident response, people skills are some of the most vital skills in the tool bag. There must be a dedicated team spirit in a CIRT when responding to critical incidents. Tempers, egos, and poor judgment cannot coexist in this type of teamwork environment. Being able to get along with team members as well as serving constituents are key elements in successfully addressing emergencies. At times, technical experts gain reputations as being difficult to work with; consequently, gathering team members with people skills can be challenging. In the arena of responding to critical incidents, team members must be adept at soothing a manager's bruised ego or an embarrassed administrator as they go about their work. Casting disparaging remarks about the employees that are responsible for day-to-day system operation certainly does not gain respect.

Incident Reporting

Along with the policy that potential or suspected critical incidents must be reported to the function-point, organizations must develop a standard for reporting emergencies that must be formalized as part of their response procedure. This procedure should include a standard checklist where critical information is elicited from the person reporting the incident.

Here is an example of a proposed incident questionnaire:

• Date of the report. Obtain from the person reporting the incident, the time, date, and place the incident was first noticed.

• Duration of the incident. How long did the incident last and what were the indications that something had happened?

• What was the name of the system being attacked?

• Where is the system located?

• What is the operating system and affected applications?

• What was the data stored on the system?

• What was the sensitivity level of the data?

• Provide a detailed description of the incident.

• How widespread is the knowledge of the attack and its details?

• What are the implications of the incident, including adverse effects on the organization?

• Incident reporter's identity, contact information, and emergency contact information for supervisor, senior manager, and system owner.

Incident reporting should be made directly to the organization's function-point that acts as the incident screener and information collector. This employee, or business unit, collects the basic information making a determination whether it should receive a formal CIRT response or be treated as a system anomaly. The information collection form might serve as the front-end of an incident database by tracking their frequency, systems affected, response posture, and improvements.

What Should I Do if I Have Been Hit?

What organizations do in the face of crisis is determined by:

• Type of critical incident

• Its impact

• Anticipated legal actions

• Best way to return to normal operations

In essence, there are two tracks to follow when responding to incidents, one requires careful and detailed coordination where evidence is collected and preserved. The other track is one guided by the overarching philosophy of "let's restore operations as soon as possible and do not worry about evidence."

Response Steps for Legal Actions

In following the "locate and prosecute to the Nth degree" track, these are the basic measures to follow:

• Determine if the emergency is a real incident. This is the most important step for the employee acting as the function-point to take. If there truly is an attack under way, immediate and decisive action is warranted, but if there is merely something developed as a result of a user-error, then administrators should be told to take appropriate action.

• If there is a qualified opinion made by the function-point, terminate attack immediately. The CIRT or a CIRT-directed effort must halt any further damage from occurring to the system's elements. There can be a lot of discussion regarding this step, but the CIRT's actions must be guided by three priorities: personnel, data, and physical facilities. Any attack affecting the confidentiality, integrity, or availability of critical assets must receive immediate attention. Given that terminating an attacker while engaged in a "live" attack will probably result in the loss of amounts of potential evidence, senior managers must decide to create policies that terminate attacks first preserving operations and worry about evidence collection as a secondary matter.

• If there has been a decision to pursue the attacker, with advice of legal counsel, law enforcement authorities must be advised as soon as possible.

• In most cases, law enforcement agencies will not assume responsibility for taking over the emergency. That obligation rests fully with the organization. Rather, officers will work with CIRT members in the investigation and collection of evidence necessary for criminal prosecutions. Depending on the agency and its policies, copies of evidence collected by officers may or may not be provided to the organization's CIRT. Make certain that there are no misunderstandings when officers arrive at the scene.

• For many departments, copies of evidence collected by law officers cannot be provided to the CIRT as a matter of policy. There are many reasons for this policy:

  • Officers collecting evidence can be compelled to testify at civil and administrative hearings where the department does not have an interest.

  • Officers may provide testimony in these proceedings that could later be used to impeach their testimony at criminal proceedings.

  • Departments do not have the resources to provide copies to the organization.

• The collection of evidence for the organization is their responsibility.

• Any legal actions taken or anticipated on the part of the organization should be coordinated with law officers. Failure to do so may have a quelling effect on their criminal prosecution and result in damage to the law officer-organization relationship.

• CIRTs must document each action taken, including the date, time, place, system name, application, operating system, and who participated. Experienced CIRT members often follow the two-employee rule.

• Any action is observed and documented by at least two persons. The reason for the two-person rule is to lessen legal challenges. All notes are considered evidentiary and must be preserved as such.

• Isolate compromised systems from the network. This is one of those initial steps limiting the proliferation of any damage. Taking systems offline is a judgment call on the part of senior managers. Depending on circumstances such as systems redundancy, equipment availability, program availability, and personnel resources, determine if this is a step where affected systems are forensically duplicated and returned to service or not. This is another one of those items to discuss with law enforcement officers as they may wish to collect the forensic copies themselves, and if the organization has qualified employees, they might be directed to create forensic copies and deliver them later to the officers.

• Discover how the attacker gained access to the affected systems. Secure the attacker's access points on all unaffected systems first, then secure the affected systems as a matter of response priorities. It is imperative that the point of attack is discovered and closed. Many times the easiest way to detect the points of entry is to compare the affected systems with "clean" systems.

• There are experts that insist on directing the attacker to a secure system where her attack process can be captured and studied. These processes are frequently known as "honeypots." While honeypots provide a lot of material for study and vulnerability analysis, their value must be weighed very carefully.

• CIRTs must document the state of the compromised systems. Maintaining a system state log is important, memorializing whether the system is in production, offline, ready to be restored to production, or replaced by a redundant system.

• Restore the victim-systems to productivity. After locating the point of entry, compare the attacked systems with the last known system-state unaffected by attacks.

  Experience Note

  Several years ago, attackers successfully invaded systems by exploiting documented vulnerabilities that were unpatched. On gaining unauthorized access, they installed backdoors, then downloaded and updated the systems. By doing this, they precluded others from invading the same systems. The organization was oblivious to the updates and the attack.

• CIRTs should document their time, resource costs, and expenditures. The cost of responding, restoring, and business resumption can form the damage-basis for civil actions in the way of estimated damages along with the cost of the equipment, revenue losses, and employee-time losses. These accumulated costs can have a significant impact during criminal trials and sentencing. Many jurisdictions establish the degree of culpability, length of sentence, and victim restitution based on costs resulting from the defendant's actions.

• CIRT members must secure all affected systems logs, audits, notes, documentation, and any other relevant evidence created or collected at the time of the incident. The evidence collection process actually has its beginning the moment the attack begins and does not cease until litigation is completed. All evidence must be documented as part of a chain of custody schedule with a copy of this document accompanying evidence-items at all times. Error on the side of caution, evidence should be catalogued on a chain of custody and even the chain of custody schedule is regarded as part of the evidence package.

• After-action briefings. This is the presentation made to senior managers where they are briefed about the incident, effects, CIRT actions, legal actions, restoration, and current systems status. In this briefing, senior managers deliver their views about CIRT's efforts, expectations, and results. At this time, it is common for CIRT's constituents to have their say. This is not the place for injured egos and hurt feelings; CIRTs should consider any and all criticisms or praise in the spirit of accomplishment or improvement.

• Postmortem. The CIRT members including full-timers, part-timers, and ad hoc members attend this meeting. Depending on the sensitivity of the discussions, outsiders who participated in the critical incident response should be in attendance. The purpose of this meeting is for CIRT members to critically analyze their performance and deliverables.

CIRT Success Metrics

The likelihood of totally eliminating attacks from outside or inside the organization is zero. CIRTs are similar to fire departments; they have significant support costs but, when activated, they are literally worth their weight in gold. Consequently, crafting a series of success metrics is usually one that is left to the very last minute. Here are a few suggestions that should be considered during the CIRT creation process:

• How many incidents did the CIRT address in a given time period? (Time periods could be measured in months, quarters, or years.)

• What were the estimated amounts of financial damage averted by CIRT intervention?

• What has been the impression of CIRT's technical expertise with their constituency?

• What is the average time and employee resources needed to address each specific incident type?

• What is the documentation completed by individual CIRT members relative to the actions taken with each incident?

• What recognition or awards were presented to the CIRT?

• Postincident feedback from constituency. Basically, this mechanism is one where a questionnaire form is provided to the victim-business unit and the results compiled by the CIRT as part of their success metrics. Particular emphasis in these questionnaires should be placed on the anonymity of the person completing them, if so desired.

• Were significant changes brought to the organization's policies and procedures suggested by the CIRT as a result of their intercession with a critical incident?

CIRT Development Life Cycle

In various forms, CIRTs have been in existence for more than 20 years. In some cases, they have performed magnificently and made substantial contributions to their organizations; while in other cases, they have foundered and sometimes failed. The levels of CIRT competence and success in the organization are tied to their development life cycle. Consequently, these are the stages of the CIRT life cycle:

• Initiation and proposal. Here is the stage where it all begins. Usually, someone makes a proposal to senior managers testing the idea and follows with a written proposal containing:

  • Necessity studies

  • Plan

  • Resource requirements

  • Structure

  • Lines of reporting and authority

  • Staffing

  • Funding

  • Training needs

  • Deliverables

  • Success metrics

  Often the employee who will serve as the unit manager begins a small ad hoc CIRT team as a pilot program. This allows the organization time to get accustomed to the concept and its execution before submitting a formal proposal. Additionally, if immediate success is realized, it makes selling the proposal much easier if a good reputation is already earned. Most employees have not heard of CIRT in this phase and do not have any expectations, yet.

• Developmental. This phase is marked by the formation of the CIRT. Much of their direction will be guided by what is done at this time. In this phase, staffing is selected or recruited, an infrastructure is created, an office site is established, equipment and tools are procured, funding is allocated, duty rosters are developed ensuring that the function-point is available to screen trouble calls at all hours, policies and procedures applicable to the CIRT are instituted, and the team is advertised as operational.

  At this stage, precedence and reputation are going to be earned. When the fledgling CIRT responds, literally every critical eye will be focused on how it performs, how it interacts with managers, and how it interacts with its constituents. Of all times, this is not the one for judgment errors or other failings. The future of the team hinges on its ability to respond quickly and bring the emergency under control with a satisfactory solution. Failing to define and obtain senior management's approval of operational requirements, drafting deficient policies and procedures, forming meaningless outside liaison contacts, and training is staff poorly can quickly spell doom for the team and its effectiveness. On the other hand, if successful the team can move on to the next stage of development.

• Establishment. In this phase startup and development problems are resolved. Constituents know when they should notify the CIRT and know what its course of action is when it arrives. In some instances, CIRTs are loaned or contracted to other organizations to assist in critical incidents. Through contracts and mutual assistance agreements, CIRTs may be deployed at business sites belonging to other organizations on a value-added basis. In this fashion, the cost of their existence is somewhat defrayed.

  In this phase, senior managers have accepted the CIRT and formally recognized its efforts. At some time in this phase, the organization and team members realize the CIRT's existence is indefinite.

  Plans are made for team progress by developing an institutional knowledge base. Team members might be considered promotions, relocation, rotation, or other work assignments. Working with the human resources unit, well-qualified prospective candidates are located and incentives provided, motivating them to consider team membership. The CIRT manager is also anxiously engaged in providing mentors for employees to upgrade training and professional certifications for her employees.

• Postestablishment. This phase includes the expansion of the team to include operations and requirements not part of any previous phases. Usually these activities include the CIRT providing constituency training, delivering presentations as guest-lecturers, authoring articles for peer-review publications, and substantial research and analysis.