Maximizing Backup Security: Veeam-Wazuh Integration Syslog
I previously wrote a blog post on custom Integration for Wazuh & Veeam.
This allowed
any rules that were triggered or matched in Wazuh from multiple sources , to initiate
a call to Veeam via the incident API this would initiate a Quick backup to
preserve data in a on-going attack & would flag any existing backup images
within the time frame as Infected or containing malicious code.
Link here :
In this
post here i take it a little bit further and we look at integrating Veeam events via
Syslog into Wazuh.
In Veeam V12.1 Veeam added syslog integration for SIEM , so that Security professionals or anyone monitoring a SOC can see events around the backup infrastructure.
Why is this important:
Malicious actors
generally target backup infrastructure to ensure organizations have no ability
to recover post the rest of their exploit / kill chain.
Additionally,
Veeam has added Inline Malware Analysis & post scanning with Yara & external
AV tools , these events can also be sent into SIEM for Visibility.
The Process: The steps I will follow in this post are as follows:
- ·
Configuring
Syslog on Veeam Server
- ·
Enabling
Syslog Listening on Wazuh Server
- ·
Testing
Syslog Integration
- ·
Creating
Decoders and Rules
- ·
Rule
Testing
- ·
Visualizing
events in Wazuh Dashboard
I supply link to the decoders I wrote & the rules I wrote which will allow
you to implement them out of the box after setting up Syslog configuration.
*Note : This
is my first pass at the rules and decoders , from the log samples I tested they
work well , but there may be some logs I haven’t tested yet , so feel free to contribute
to the rules and decoders or drop me a comment to PR on Github.
Link to My Github Repo for this post: mritsurgeon/wazuh: Decoders & Rules (github.com)
Step 1 :
Veeam
In Veeam we access
the VBR Console ( Veeam backup & Replication Console )
Select the burger menu (Top Left ) – Select Options – Select SIEM Integrations Tab – & Add SYSLOG Server
Configure Syslog on Veeam server – This is the IP of Wazuh SIEM , I used TCP and Default Port
On Clicking
Apply – You are presented with and Error & this is because you have not enabled
Syslog listening on Wazuh server. This is the next step.
The Error:
You then need to
SSH to Wazuh Server & we need to enable the config on the WAZUH server to
allow SYSLOG from Veeam :
Example
config :
Add the
following configuration in between the <ossec_config> tags of the Wazuh
server /var/ossec/etc/ossec.conf file to listen for syslog messages on TCP port
514:
Configuration
Below :
Where:
<connection>
specifies the type of connection to accept. This value can either be secure or
syslog.
<port>
is the port used to listen for incoming syslog messages from endpoints. We use
port 514 in the example above.
<protocol>
is the protocol used to listen for incoming syslog messages from endpoints. The
allowed values are either tcp or udp.
<allowed-ips>
is the IP address or network range of the endpoints forwarding events to the
Wazuh server. In the example above, we use 192.168.2.15/24.
<local_ip>
is the IP address of the Wazuh server listening for incoming log messages. In
the example above, we use 192.168.2.10.
More
information on XML Tags here : remote
- Local configuration (ossec.conf) · Wazuh documentation
sudo vi /var/ossec/etc/ossec.conf
Add the
Syslog settings between the <ossec_config> tags of the Wazuh server
Important for Debugging : Additionally add this config to the global settings so that we catch all logs including those Wazuh does not understand :
ESC then : wq!
To save the changes in VI
Finally restart
the Wazuh manager to apply the changes:
systemctl
restart wazuh-manager
Once Completed: Apply Syslog setting in Veeam , You should no longer get an error :
All Syslog
events from VBR are now being sent to Wazuh Archive Log :
If we CAT
the log we can find a test Syslog MSG sent from veeam.
Great so we
are getting syslog Msgs from Veeam , the challenge now is that Wazuh does not
understand what the log is for or what it contains, we need to follow 2 steps
here.
Firstly we
need to Decode the logs using general layout or format the veeam logs will
arrive in , so that we can extract data as relevant fields.
We then
will create rules that we will use extracted field data to match criteria , these rules
will trigger as events in Wazuh Dashboard.
Here is a
better explanation of the process to match data in log to a rule or event :
Wazuh processes logs through several phases,
which are essential for accurate log analysis and rule matching. The phases
are:
1. Pre-decoding Phase: In this phase,
the raw log data is processed but not yet decoded. This phase involves tasks
such as identifying the log source, extracting timestamps, and performing any
necessary pre-processing steps.
2. Decoding Phase: During this phase,
Wazuh attempts to match the pre-processed log data against configured decoders.
Decoders define patterns and regular expressions to extract relevant fields and
metadata from the log data. If no decoder matches the log data, Wazuh proceeds
to the next phase without decoding the log.
3. Rule Matching Phase: In this final
phase, Wazuh applies the defined rules against the decoded log data. Rules
consist of conditions and actions, allowing Wazuh to generate alerts, execute
commands, or perform other actions based on the log data.
Creating
Decoders
Ok so let’s copy the test MSG we found in the archive log :
2024 Mar 30
10:56:52 wazuh->172.21.234.6 1 2024-03-30T08:01:18.838212-07:00 VBR
Veeam_Backup - - [origin enterpriseId="31023"] This is a test syslog
message from Veeam Backup & Replication
Then we login to
Wazuh console :
Next , Lets run a
Rule test : we going to see if Wazuh can decode the Syslog msgs from Veeam
& then trigger a rule
This can be done from CLI or UI – go to Ruleset test – under tools in Wazuh console:
Alternatively you can run /var/ossec/bin/wazuh-logtest from the command line.
We run a
test on the test msg from Veeam syslog.
Phase 1: This
phase involves pre-decoding, where the raw log data is processed but not yet
decoded.
Phase 2:
This phase involves decoding the pre-processed log data. However, in this case
we can see that no decoder matched the
log data because there is no appropriate decoder configured for it.
So we need
to write decoders for Veeam logs :
On the Wazuh
server in /var/ossec/etc/decoders
There is an
existing local_decoders.xml file that you can edit or add too , but I added
roughly 27 decoders so I created a new file Veeam_decoders.xml
The easiest field to extract is Veeam Event Ids from instanceId= , you can then match this against veeam events.
Here is a list of Veeam events : Veeam Backup & Replication Events - User Guide for VMware vSphere
I have extracted all the Event IDs and added them to a CSV for easier processing:
wazuh/Veeam Event IDs.csv at main · mritsurgeon/wazuh (github.com)
Example decoder:
This will fetch the instance Id ( Event ID ) and extract it under a field called EventID which we can then use to match in rules.
However, we might want to want more information. Example: User, Date or Time etc.
So, we add more decoders to extract relevant fields.
XML Example:
Here is a link to my pre-created decoders: wazuh/Veeam_Decoders.xml at main · mritsurgeon/wazuh (github.com)
Lets explain the the decoders
High-level
Breakdown of the Veeam Decoders:
The Veeam
decoders are designed to extract various fields and metadata from Veeam Backup
& Replication logs.
Veeam MP
Decoder: This
decoder acts as a parent decoder, using a regular expression to match the log
message format and extract the enterprise ID, category ID, and instance ID.
** This is important because we don’t want to send all logs to these list of decoders , so we qualify what is a relevant Veeam log using persistent entries that appear in each log to make it a Veeam type log.
General
Decoders: These
decoders are firstly child decoders that inherit from the Veeam MP decoder.
They use regular expressions to extract specific fields from the log data, such
as user names, VM names, job names, descriptions, states, and various other
relevant information.
This approach
allows for efficient and organized log processing, ensuring that Wazuh can
extract the necessary fields and metadata from Veeam logs for further rule creation.
The General
Decoders are also sibling decoders , because similar logs can differ , for
example the field $User could be matched using different regex techniques across
multiple logs based on its positioning or sequence in which its displayed in
the logs , so we need similar values but from different Regex – the approach
here is sibling decoders. ( which basically means they all have the same name )
Here is a Diagram
& a Link to Wazuh documentation that might explain it a bit better : Sibling
Decoders - Decoders · Wazuh documentation
If you want to modify the decoders , I relied on multiple tools to test my regex , but keep it simple as possible , you can use regex101: build, test, and debug regex against the log & see what it matches , what’s important , under match information you have match 1, 2, 3 etc , this will be your field data that you extract for rules.
So you can get my
decoder file , sharing it again here wazuh/Veeam_Decoders.xml at main · mritsurgeon/wazuh (github.com) and add it
to /var/ossec/etc/decoders on Wazuh server.
Restart the dashboard services again: systemctl restart wazuh-manager
You can
then re-test another log , for instance this log :
2024-03-30T08:26:11.635070-07:00 VBR Veeam_MP - - [origin enterpriseId="31023"] [categoryId=0 instanceId=10050 OibID="ea3dd562-c1a4-4f14-8f8a-c9a332b42a05" OriginalOibID="ea3dd562-c1a4-4f14-8f8a-c9a332b42a05" VmRef="1e841042-351f-4a43-dba9-b7f7bc8de8eb" VmName="target" ServerName="This server" DateTime="03/30/2024 15:26:11" IsCorrupted="True" Platform="6" StorageSize="1617920" RepositoryID="07df5866-8e8b-4326-a5fa-f0b8585dd64b" IsFull="False" Version="1" Description="Restore point for VM 'target' has been removed by user SYSTEM."]
You can see
now that the Phase 2 has completed the decoding successfully & that data
is extracted and assigned to fields.
We can use these fields to trigger rules, for example : Object that Equals Restore Point with the State Removed – could trigger an alarm to indicated data is being deleted in backup.
Or
We can use field Event ID to match for this rule IE: EventID: '10050'
Note : In my GitHub Repo you will find 2 rules files:
Generic Rules XML:
In the generic rules XML file, you'll find rules that cover all event IDs generated by Veeam. These rules are designed to capture and categorize events broadly, without focusing on specific details within each event. Here's how each rule is structured:
Rule ID: A unique identifier for the rule.
Level: The severity level assigned to events captured by this rule.
Decoded as: Indicates the source or type of the event Decoder, in this case, Veeam MP.
Description: A placeholder for the description of the event.
Field: Specifies a field within the event data that the rule is targeting. Here, it's filtering events based on the "EventID" field having the value "28700".
These generic rules provide a baseline for capturing all Veeam events, which can be further customized and extended as needed.
Field-specific Rules XML:
In the field-specific rules XML file, you'll find rules that utilize specific fields within Veeam events to provide more detailed and targeted analysis. These rules focus on correlating events based on certain field values. Here's an example:
Rule ID, Level, Decoded_as, and Description are similar to the generic rules.
Field: Specifies the fields within the event data that the rule is targeting for correlation.
Here, it's looking at the "Object" field with the value "Backup" and the "State" field with values matching the regular expression pattern "^mounted." or "unmounted."
These field-specific rules offer more granular control over event analysis by focusing on specific attributes within Veeam events. You can use these rules as templates for creating custom correlations and alerts tailored to their specific environment and requirements.
Note: In Field-specific Rules XML , you will find Custom Groupings
Here's a summary of rule groups , you can move rules between groups and change the severity level based on preference.
Veeam_Backup_Replication_v12.1 Info: This group contains rules with a level of 4 (informational), covering events such as backup mounts, started sessions, scan backups, console launches & successful jobs or tasks.
Veeam_Backup_Replication_v12.1 Warning: This group includes rules with a level of 6 (warning), monitoring events like backup repository modifications, four-eyes policy changes, job warnings, malware detection events, and retention value changes.
Veeam_Backup_Replication_v12.1 Error: This group contains rules with a level of 8 (error), focused on critical events such as restore point removals, failed jobs, VM or agent deletions, and other potentially harmful actions.
Veeam_Backup_Replication_v12.1 Potential IOC: This group includes rules with a level of 12 (critical), designed to detect potential indicators of compromise (IOCs) like malware detection events, failed malware detection sessions, and clean restore point unavailability.
The rules leverage regular expressions, field matching, and specific keywords to identify and categorize different types of events from Veeam logs. Additionally, some rules incorporate MITRE ATT&CK techniques, making it easier to correlate events with known attack patterns and enhance incident response.
For Example : MITRE ATT&CK ID T1490 to Inhibit System Recovery, when backup data is removed by a User. I Also added for GDPR and other Governance groupings to rules like Failed backup & change in retention.
The 2 Rule files in the Github Repo
Field_Veeam_Rules.xml - Some Custom Rule Examples with MITRE IDs
wazuh/Field_Veeam_Rules.xml at main · mritsurgeon/wazuh (github.com)
Generic_Veeam_rules.xml - Full Veeam Event ID List
wazuh/Generic_Veeam_rules.xml at main · mritsurgeon/wazuh (github.com)
Writing rules
This Brings
us to Phase 3 the Rules :
Again in /var/ossec/etc/rules
on the Wazuh server there is a Local_Rules.xml that you can add too but because
I added a large amount of rules & i have 2 versions of how rules can be implemented , I chose to create a separate files as shared above.
Once rules are saved to waxuh server as XML under /var/ossec/etc/rules , we need to restart Wazuh Dashboard services.
systemctl restart wazuh-manager
Testing the
rules :
Once the
rules are added we can navigate back to ruleset testing in Wazuh console &
test a log entry , I used this log snippet :
2024-03-31T12:22:55.638476-07:00 VBR Veeam_MP - - [origin enterpriseId="31023"] [categoryId=0 instanceId=42290 param1="0" param2="0" param3="0" param4="0" param5="<changes><object /></changes>" param6="VBR\Administrator" Version="1" Description="Malware detection settings have been changed."]
You can see
it passes through Phase 1 & 2 getting decoded properly and then its matched
by a rule that includes a potential Mitre Attack id , tactic & technique. This is against the Field_Veeam_Rules.xml file.
This is a
great way to test & debug rules & decoders, if you have any missing information
in alerts, you can test the log sample here and see what is missing, its either a decoder not extracting information you need or there is no rule that matches the ID or fields used.
Finally , you can generate some alerts by making some changes in Veeam and then revisiting
the Wazuh dashboard to view the events or rules triggered,
I found it easiest
to filter Security events by decoder.name & matching it against Veeam MP to see all events
that were processed by the Veeam decoders. You can also filter by any of the fields that you extracted during decoding. IE : We could filter all Security events by $User field.
We can see events
starting to come through on the Wazuh Dashboard & you can continue to customize & fine tune views , filters or the rules & decoders themselves.
In conclusion:
This was my
first pass or V1 of the Veeam Wazuh decoders & rules & I did not test against the entire Veeam log set, but across the 26 decoders & few 100 rules this should match the majority of logs
, however you might want to change what fields you extract based on how you want your
rules to trigger & what you want to be alerted on. Furthermore you would need to change the severity of the rules based on what is pertinent to your security team to be alerted on, there are examples of this customization in the Field_Veeam_Rules.xml file.
Thank you
for reading this far , please comment & share.
Comments
Post a Comment
Leave your Thoughts & Comments , and I'll reply as soon as possible.
Thank you for you views in Advance.