Log Collection
Log collection can be a daunting task, especially if massive amounts of data need to be collected. This process could be made easier by previously considering some key points.
Choosing Log Event Sources
The selection of event sources should be driven by the use cases that have been identified. For example, if the use case is the monitoring of user logins to critical resources, the event sources selected should be only those related to the critical resources in question. This may include the LDAP directory server, local servers, firewalls, network devices, and key applications.
There are many other potential event source categories, including:
Security
Firewalls
Endpoint Security (EDR, AV, etc.)
Web Proxies/Gateways
LDAP/Active Directory
IDS
DNS
DHCP
Servers
Workstations
Netflow
Ops
Applications
Network Devices
Servers
Packet Capture/Network Recorder
DNS
DHCP
Email
DevOps
Application Logs
Load Balancer Logs
Automation System Logs
Business Logic
Collection Methods
A decision must be made as to how the logs will be collected. After a list of event sources has been determined, the next step is to decide the method of collection for each source. It is critical to understand what method each event source uses and what resources may be required.
For example, if a log shipper will be required to read logs from a local file on all servers, the log shipper must be selected and tested before deployment. In other cases, proprietary APIs or software tools must be employed and integrated. In some cases, changes to the event sources themselves (security devices, network hardware, or applications) may be required. Additional planning is often required to deploy and maintain these collection methods over time.
Last updated