Application logging for security purposes
13 Apr 2020
Finding useful information in application logs can be a challenge. Most applications logging is focused on logging exceptions, proper logging however should also make it possible to get insight in important transactions processed by the application. Not only focusing on exceptions encountered but also on things like throughput and (non-exception) anomalies. But application logs can also play a major role in the security of an application. This blogpost will focus on how to use your application logs for security purposes. There is much more to tell about improving logs but that might be something for another post.
Reasons to use logging for security purposes
Proper logging can be used to increase the security of your application in a lot of ways. Some common security uses for application logs are lister below:
- Proactive detection of suspicious activity
- Detailed analysis after a security breach
- For audit purposes
- Find trends which could indicate malicious intent
What events to log
Many types of application events can be useful for security reasons. The most common events, and the ones you should definitely log are the following:
- Input events
- Input validation failures
- Attempts to add extra input parameters
- Authentication events
- Logins to the application with invalid credentials
- Logins with invalid credentials from the application to another service or system
- Creation of new accounts
- Logins with admin accounts
- Authorization events
- Unauthorized access attempts
- Authorized use of high-risk functionality (like payments, orders, changes to accounts)
- All actions by admin accounts which change settings or data
- Request events
- Requests to non existing resources (could be an attempt to scan for vulnerable endpoints)
The list is by no means complete. Analyze your application and determine which functionality or user actions are most important and how these could be exploited. Then add extra logging with enough details to allow you to determine if and how the functionality was exploited.
What details to log
As with all types of log entries, you'll want to record the 'when, where, what and who' in your security logs. You have to pay extra attention to recording the who. For analysis, and threat detection it is vital to know which user or system initiated the event you logged.
- Exact (server) time of the event
- Location from where the event was recorded (endpoint, URL, script method, ...)
- Details about the origin from where the event was initiated (IP, geolocation, origin system details like useragent, ...)
- Description of the (type of) event (input validation, failed login, etc.)
- The user or system who initiated the event
- Any extra data needed to determine what happened :
- The input which caused the validation failure
- The changes made when high-risk functionality was used
What not to log
You should include just enough data in your security logs to allow you to detect suspicious actions. But logging too much details can lead to exposure of sensitive data in your logs which could lead to security vulnerabilities. So be very careful not to include sensitive data in your log entries. Especially try to avoid the following data:
- Passwords
- Personally Identifiable Information
- Bank account or credit card details
- Database connection details
- Source code details
Separating security from regular log entries
Security application log entries can be just an integral part of your overall application logs, whether these logs are stored in files, Elastic or any other datastore. But in some cases you'll want to treat the security entries in your logs differently. Common reasons why you'll want to do is are:
- Security log entries must be stored longer than other log entries (for audit reasons it may be compulsory to store certain security log entries, like user access, for a very long time)
- Security log entries may contain sensitive data, so access to them should me more restricted than to other log entries
- Security log entries must be sent to a security monitoring tool (like a SIEM) so you must be able to filter them out of your logs easily
To be able to distinguish your security and regular log entries there are two main options:
- Adding a category or tag to your security log entries to allow easy distinguish your security log entries from your other log entries
- Setting up your application logging in such a way that all security log entries are stored in a separate store (file, Elastic, Database or other) from the rest of your log entries
Which option to use depends mainly on your application and logging infrastructure. Do you log to file or to a datastore? Does your application logging system allow logging to multiple datastores? Wat options does your logging system have for filtering?
Examples
All log entries listed below are just examples of what kind data could be added to make the log entries useable for security purposes. They should just be used as examples, always think carefully about what data you'll need in your specific scenario, and if you have enough options to filter out the details you need in the logging dashboard you are using.
Log entry for input validation failure
This example show a data set which can be logged to help identify a validation failure. With the data logged for this example we can determine when the event took place, who was responsible (user 123456 from IP 123.123.123.123) , where it took place (URL /form) and what happened (invalid characters for field 'Name' and InputData was <script>). Finally a 'security' tag was added to make it easy to filter out this log entry in a security logging dashboard.
| Timestamp | Message | Tags | Url | IP | UserId | InputData |
|---|---|---|---|---|---|---|
| 2020-02-26T17:21:53 | Input validation failure : invalid characters for field 'Name' | security | /form | 123.123.123.123 | 123456 | <script> |
Log entry for login with invalid credentials
The following example again uses the IP and UserId (if this can be determined from the login attempt) to identify the origin of the action (the who). The what is a combination of the Message field and the InputData field. Logging usernames can be tricky from a privacy point of view, especially if the username is an email address. But in the case of a failed login attempt logging the username/email used in the login attempt may be acceptable. Logging the password (or other secret used in the login) is rarely a good idea, so leave it out of your logging.
| Timestamp | Message | Tags | Url | IP | UserId | InputData |
|---|---|---|---|---|---|---|
| 2020-02-26T17:25:53 | Login with invalid credentials. | security | /login | 123.123.123.123 | 123456 | user@user.nl |
Next steps
Logging the right events with the right details is just the first step in setting up proper security logging. You'll have to think about where to store your security logs, do you use the same store as for your regular logs or a dedicated security log store. What kind of dashboards will you need to access your security logs and who will have access to those dashboards. How long do you have to store your security logs, will you have to store them longer than your regular application logs? And last but not least, you'll want to think about how to add monitoring and alerting. To often logging is only looked at after something goes horribly wrong. Being alerted before something is about to go horribly wrong is the ultimate usage of proper logging.