Sample Information Security – Incident Handling guideline

Incident Handling
General
This document provides guidelines and procedures for dealing with computer security incidents. The document is meant to provide support personnel with some guidelines on what to do if they discover a security incident.
A computer security incident can occur at anytime of the day or night.  Most hacker/cracker incidents occur during the off hours when hackers do not expect system managers to be watching their flocks. However, worm and virus -incidents can occur any time during the day, and many security incidents are at the hands of employees. Thus, time and distance considerations in responding to the incident are very important. If the first person on the call list to be notified can not respond within a reasonable time frame, then the second person must be called in addition to the first. It will be the responsibility of the people on the call list to determine if they can respond within an acceptable time frame.

The media is also an important consideration. If someone from the media obtains knowledge about a security incident, they will attempt to gather further knowledge from a site currently responding to the incident. Providing information to the wrong people could have undesirable side effects.

Definitions

The term “incident” refers to any situation that deals with a potential breach or violation of computer, network or information security. Some examples of incidents include:

· When information assets are accessed in an unspecified or unauthorized manner, including unauthorized use of an account, accessing files, directories or applications which one has not been given explicit access to, exploiting known vulnerabilities, hacking into systems, etc.

· When an information asset is used for purposes other than were intended, such as accessing sexually explicit internet sites, using privileged accounts beyond the scope of the intended responsibility, etc.

· When a user or program disrupts network or computer services by denying service to them, or badly degrading normal service.

Responsibility
Every employee is responsible for reporting an incident or potential security incident.  In many cases, a single person on a single system will not perform the actions outlined in this guideline. Many people may be involved during the course of an active security incident which affects several of the systems at one time (i.e., a worm or virus attack). The Information Security Department should always be involved in the investigation of any security incident; they will act as the incident coordination team for all security related incidents.
The incident coordination team will be responsible for assigning people to work on specific tasks of the incident handling process and will coordinate the overall incident response process. All people involved in the incident response and clean up are responsible for providing any needed information to members of the incident coordination team.
Any directives given by a member of the incident coordination team will supersede this document.

General Procedures

Keep a Log Book

Logging of information is critical in situations that may eventually involve federal authorities and the possibility of a criminal trial. The implications from each security incident are not always known at the beginning of, or even during, the course of an incident. Therefore, a written log should be kept for all security incidents that are under investigation. The information should be logged in a location that can not be altered by others. Manually written logs are preferable since on-line logs can be altered or deleted. The types of information that should be logged are:

· Dates and times of incident-related phone calls.
· Dates and times when incident-related events were discovered or occurred.
· Amount of time spent working on incident-related tasks.
· People you have contacted or have contacted you.
· Names of systems, programs or networks that have been affected.

Inform the Appropriate People

Informing the appropriate people is of extreme importance. There are some actions that can only be authorized by the Site Security Manager or Information Security Manager. also has the responsibility to inform other sites about an incident that may effect them. The Information Security Department maintains call agreements with both the ON-Line Solutions Helpdesk and the Site Security Department.  During normal operating hours the first course of action should be to call the ON-Line Solutions Helpdesk to report the incident.  Outside of normal operating hours, the Site Security Department should be notified.

Release of Information

Control of information during the course of a security incident or investigation of a possible incident is very important. Providing incorrect information to the wrong people can have undesirable side effects, especially if the news media is involved. All release of information must be authorized by the Information Security Manager or by the Public Relations Department. All requests for press releases must be forwarded to the Information Security Department.  Also, incident specific information, such as accounts involved, programs or system names, are not to be provided to any callers claiming to be a security officer from another site. All suspicious requests for information should be forwarded to the Information Security Department. If there is any doubt about whether you can release a specific piece of information contact the Information Security Department.

Follow-Up Analysis

After an incident has been fully handled and all systems are restored to a normal mode of operation, a follow-up postmortem analysis should be performed. The follow-up stage is one of the most important stages for handling a security incident. All involved parties (or a representative from each group) should meet and discuss actions that were taken and the lessons learned. All existing procedures should be evaluated and modified, if necessary. All on-line copies of infected files, worm code, etc., should be removed from the system(s). If applicable, a set of recommendations should be presented to the appropriate management levels. A person designated by the Information Security Manager should write and distributed a security incident report to all appropriate personnel should write.

Incident Specific Procedures

This section discusses the procedure for handling virus, worm and hac
ker/cracker incidents.

Virus and Worm Incidents

Although virus and worm incidents are very different, the procedures for handling each are very similar.  Most security incidents are time critical.  Worms and Viruses can be self-replicating, spreading to hundreds of machines in a matter of minutes.  Often a virus will be identified and destroyed by an installed Virus scanner, but it is still important to contact the ON-Line Helpdesk and report the virus sighting.  The following procedure should be followed if a virus is identified, but not destroyed, or if you suspect a virus infection.  The following information is divided into two sections; the first is general procedures, and then technician specific procedures.

General

1) Isolate the System
Isolate infected system(s) from the remaining network as soon as possible.  Network isolation is one method to stop the spread of a worm or virus, but the isolation can also hinder the clean up effort since will be disconnected from sites or resources which may have patches. The Director of Network Infrastructure must authorize the isolation of the network from the outside world.

2) Log all actions
Do not power off or reboot systems that may be infected. There are some viruses that will destroy disk data if the system is power-cycled or rebooted. Also, rebooting a system could destroy needed information or evidence.

3) Notify Appropriate People

Notify the InfoSec Helpdesk as soon as possible.  The Helpdesk will assess the situation and either perform remote inoculation and analysis, or will dispatch a technician.

Technicians

1) Identify the Problem

Try to identify and isolate the suspected virus or worm-related files and processes. Prior to removing any files or killing any processes, a snapshot of the system should be taken and saved.  A system snapshot is essentially a listing of the current system configuration and operating parameters, system & security logs, and processes operating on the machine.

If specific files that contain virus or worm code can be identified, then move those files to a safe place or archive them to tape and then remove the infected files. Also, get a listing of all active network connections. Virus software vendor’s technical support can provide assistance in obtaining snapshot information on the system, for diagnosing and creating an inoculation for the virus.

If other sites have been involved at this point, they may have helpful information on the problem and possible short-term solutions. Also, any helpful information gained about the virus or worm should be passed along to Internet CERT sites, after approval by Information Security Manager.  Log all actions.

2) Contain the virus or worm

All suspicious processes should now be halted and removed from the system. Make a full dump of the system and store in a safe place. The tapes should be carefully labeled so unsuspecting people will not use them in the future. Then remove all suspected infected files or worm code. In the case of a worm attack, it may be necessary to keep the system(s) isolated from the outside world until all systems have been inoculated and/or the other Internet sites have been cleaned up and inoculated.  Log all actions.

3) Inoculate the System(s)

Implement fixes and/or patches to inoculate the system(s) against further attack. Prior to implementing any fixes, it may be necessary to assess the level of damage to the system. If the virus or worm code has been analyzed, then the task of assessing the damage is not very difficult. However, if the offending code has not been analyzed, then it may be necessary to restore the system from backup tapes. Once the system is brought back into a safe mode, then any patches or fixes should be implemented and tested. If possible, the virus or worm should be let loose on an isolated system that has been inoculated to ensure the system(s) are no longer vulnerable. Log all actions.

4) Return to a Normal Operating Model

Prior to bringing the systems back into full operation mode, you should notify the same group of people who were notified in stage one. The users should also be notified that the systems are returning to a fully operational state. It may be wise to request all users to change their passwords. Before restoring connectivity to the outside world, verify that all affected parties have successfully eradicated the problem and inoculated their systems. Log all actions.

5) Follow-up Analysis

Perform follow-up analysis as described in the General Procedures section.

Hacker/Cracker Incidents

Responding to hacker/cracker incidents is somewhat different than responding to a worm or virus incident. Some hackers are very sophisticated and will go to great depths to avoid detection. Others are naive young students looking for a thrill. A hacker is often someone on the inside engaging in illicit system activity (i.e., password cracking). Any hacker/cracker incident needs to be addressed as a real threat to the information infrastructure.

Hacker incidents can be divided into three types: attempts to gain access to a system, an active session on a system, or events, which have been discovered after the fact. Of the three, an active hacker/cracker session is the most severe and must be dealt with as soon as possible.

There are two methods for dealing with an active hacker/cracker incident. The first method is to immediately lock the person out of the system and restore the system to a safe state. The second method is to allow the hacker/cracker to continue his probe/attack and attempt to gather information that will lead to an identification and possible criminal conviction. The level of understanding of the risks involved will determine the method used to handle a cracker/hacker incident.  Regardless, Information Security should always be contacted first to assess the situation and coordinate the proper method of dealing with the incident.  The decision to immediately lock the person out of the system should only be done without consulting Information Security if there is an immediate and catastrophic threat the integrity of critical data or systems.

Attempted Probes into a System

Incidents of this type would include repeated l
ogin attempts, repeated ftp, telnet or rsh commands, repeated dial-back attempts, and unsolicited network scans.

1) Identify Problem

Identify source of attack(s) by looking at system log files and active network connections. Make copies of all audit trail information such a system logs files, the root history file, the utmp and wtmp files, and store them in a safe place. Capture process status information in a file and then store the file in a safe place. Log all actions.

2) Notify Information Security

Notify the Information Security Department within 30 minutes. If an InfoSec Analyst can not be reached then notify the InfoSec Manager or their designated backup personnel. The InfoSec Analyst or their backup person will be responsible for notifying other levels of management.

3) Identify Hacker / Cracker

If the source of the attacks can be identified, then the InfoSec Analyst (or a designated person) will contact the system administrator or security analyst for that site and attempt to obtain the identity of the hacker/cracker. IP ownership can be determined from the regional IP registrar, for the USA it’s the American Registry of Internet Numbers (ARIN, http://www.arin.net).

Domain ownership and contact information can be enumerated from an accredited Domain Registrar.  If the hacker/cracker can be identified, the information should be provided to the InfoSec Analyst coordinating the incident response. They will provide directions on how to proceed, if necessary. Log all actions.

4) Notify CERT

If the source of the attacks can not be identified, then the Analyst will contact the Internet CERT, CIAC, or InfraGuard teams and provide them with information concerning the attack, as authorized by the InfoSec Manager.

Note:  The InfoSec Manager or someone he designates must approve release of information. Log all actions.

5) Follow-up

After the investigation, a short report describing the incident and actions that were taken should be written by the InfoSec Analyst or Manager and distributed to the appropriate people. Perform the follow-up analysis as described in the General Procedures Section.

Active Hacker/Cracker Activity

Incidents of this type would include any active session or command by an unauthorized person. Some examples would include an active rlogin or telnet session, an active ftp session, or a successful dial-back attempt. In the case of active hacker/cracker activity, a decision must be made whether to allow the activity to continue while you gather evidence or to get the hacker/cracker off the system and then lock the person out. Since a hacker can do damage and be off the system in a matter of minutes, time is critical when responding to active hacker attacks. The InfoSec Manager or someone he designates must make this decision.  The decision will be based on the availability of qualified personnel to monitor and observe the hacker/cracker and the level of risk involved.

1) Notify Appropriate People

Notify the Information Security Department as soon as possible.  The Information Security Department maintains escalation agreements with both the Helpdesk and Site Security Teams.  Helpdesk staff will route InfoSec calls to the on-call analyst.  If unable to reach him/her within 5 minutes, they will contact the backup person. The InfoSec analyst will then be responsible for notifying other appropriate personnel. The analyst will be responsible for trying to assess what the hacker/cracker is after and the risks involved in letting the hacker/cracker continue his/her activity.

The analyst will notify the Information Security  Manager as soon as possible. If unable to reach him within ten minutes, his backup person should be contacted. The Information Security Manager can make the decision to allow the hacker to continue or to lock him out of the system. Based on the decision, follow the General Procedures.

Should the incident require further escalation, to the Director of Network Infrastructure or the CIO, the Information Security Manager will contact them within 30 minutes.

Removal of Hacker/Cracker from the System

1) Snap-shot the System

Make copies of all audit trail information such as system logs files, the root history files, access record files such as utmp and wtmp, and store them in a safe place. Capture process status information in a file and then store the file in a safe place. Any suspicious files should be moved to a safe place or archived to tape and then removed from the system. Also, get a listing of all active network connections. Log all actions.

2) Lock Out the Hacker

Kill all active processes for the hacker/cracker and remove any files or programs that he/she may have left on the system. Change passwords for any accounts that were accessed by the hacker/cracker. At this stage, the hacker/cracker should be locked out of the system. Log all actions.

3) Restore the System

Restore the system to a normal state. Restore any data or files that the hacker/cracker may have modified. Install patches or fixes to close any security vulnerabilities that the hacker/cracker may have exploited. Inform the appropriate people. All actions taken to restore the system to a normal state should be documented in the logbook for this incident. Log all actions.

4) Notify Other Agencies

Report the incident to the InfraGuard, the Internet CERT or to CIAC.

Note:  the InfoSec Manager or someone he designates must approve Release of information. Log all actions.

5) Follow-up

After the investigation, a short report describing the incident and actions that were taken should be written by the InfoSec Analyst or Manager and distributed to the appropriate people. Perform the follow-up analysis as described in the General Procedures section.

Monitoring of Hacker/Cracker Activity

ON InfoSec staff maintains set procedures for monitoring of illicit activity on certain systems, although often there may be no procedures for the system in question. Each incident will be dealt with on a case by case basis. The InfoSec Manager or the person authorizing the monitoring activity should provide direction to those doing the monitoring. Once the decision has been made to cease monitoring the hacker’s activities and have him removed from the system(s), the steps outlined in the Hacker/Cr
acker Removal section above should be followed.

Evidence of Past Incidents

In the case of where an incident is discovered after the fact, there is not always a lot of evidence available to identify who the person was or how they gained access to the system. If you should discover that someone had successfully broke into an system, notify the Information Security Department within one working day. The Information Security Department will be responsible for notifying the appropriate people and investigating the incident.

Recovering from Past Incidents

Often in the case of discovering past incidents it is advisable to follow standard recovery procedures, and regress system states back to a verifiably secure state, even to the extent of completely rebuilding the machine in question.  Although this is a worst case scenario, often should a system be fully compromised and discovered after the fact, this is the only course.