Splunk Field Extraction Walkthrough
In this video I will cover different ways to parse data that you may have already ingested into Splunk. I will walkthrough on how to use Splunk's internal field extractor and then will dive into using the props.conf & transforms.conf when advance field extraction is required.
You will also see me use the website - https://regex101.com/
Regex101 is a great site to help build out your regular expressions which I have used on numerous occasions.
My plan is to build a series of videos that I will call "How Travis does stuff in Splunk".
Here is the content of the transforms.conf I used in this video. I will mention that "dpkg_installed" was incorrect in the video and has been corrected below.
This will go in your transforms.conf, do not copy the props.conf section below:
[dpkg_install]
REGEX = (install)\s+(.+):(.+)\s+(.+)\s+(.+)
FORMAT = action::$1 package::$2 info::$3 none::$4 version::$5
[dpkg_installed]
REGEX = (installed)\s+(.+):(.+)\s(.+)
FORMAT = action::$1 package::$2 info::$3 version::$4
[dpkg_half_installed]
REGEX = (half-installed)\s+(.+):(.+)\s(.+)
FORMAT = action::$1 package::$2 info::$3 version::$4
[dpkg_half_configured]
REGEX = (half-configured)\s+(.+):(.+)\s(.+)
FORMAT = action::$1 package::$2 info::$3 version::$4
[dpkg_unpacked]
REGEX = (unpacked)\s+(.+):(.+)\s(.+)
FORMAT = action::$1 package::$2 info::$3 version::$4
[dpkg_configure]
REGEX = (configure)\s+(.+):(.+)\s+(.+)\s+(.+)
FORMAT = action::$1 package::$2 info::$3 version::$4 none::$5
[dpkg_upgrade]
REGEX = (upgrade)\s+(.+):(.+)\s+(.+)\s+(.+)
FORMAT = action::$1 package::$2 info::$3 version_old::$4 version_new::$5
[dpkg_remove]
REGEX = (remove)\s+(.+):(.+)\s+(.+)\s+(.+)
FORMAT = action::$1 package::$2 info::$3 version::$4 none::$5
[dpkg_trigproc]
REGEX = (trigproc)\s+(.+):(.+)\s+(.+)\s+(.+)
FORMAT = action::$1 package::$2 info::$3 version::$4 none::$5
[dpkg_startup]
REGEX = (startup)\s+(.+)\s+(.+)
FORMAT = action::$1 type::$2 status::$3
-----------------------------------------------
Here is 2 different ways to use the props.conf to call out the transforms.conf file and parse the data. I have included notes about each method and you will need to determine what is best for you. In the video I demonstrated the 2nd method. Please only use 1 method.
#Using the below does not work on data that has been indexed already. This will be for any new data for cleaner looking sourcetype if that is a concern.
Use this in your props.conf:
[source::.../dpkg.log]
sourcetype = dpkg
[dpkg]
REPORT-dpkg_all = dpkg_install, dpkg_installed, dpkg_half_installed, dpkg_half_configured, dpkg_unpacked, dpkg_configure, dpkg_upgrade, dpkg_remove, dpkg_trigproc, dpkg_startup
#If the data has been indexed, the following method will parse at search time.
Use this in your props.conf
[source::.../dpkg.log]
REPORT-dpkg_all = dpkg_install, dpkg_installed, dpkg_half_installed, dpkg_half_configured, dpkg_unpacked, dpkg_configure, dpkg_upgrade, dpkg_remove, dpkg_trigproc, dpkg_startup
-----------------------------------------------
As always don't forget to check out my other videos and GoSplunk where I have posted some of my dashboard and queries built over the years.
https://gosplunk.com/author/thall/
travis.