Where does data go once ingested into Splunk? 

Does Splunk use files and folders?

How Splunk Stores Data

In Splunk data is stored into buckets. Not real bucket filled with water but buckets filled with data. A bucket in Splunk is basically a directory for data and index files. In a Splunk deployment there are going to be many buckets that are arranged by time. In this video learn the 5 types of buckets in Splunk every administrator should understand.

Transcript – 5 Types of Buckers in Splunk

Hi folks! Thomas Henson here with thomashenson.com. Today is another episode of Big Data Big Questions. Today’s we’re going to be talking about the five different kind of buckets in Splunk. We’re going to go through, we’re going to talk about how Splunk uses buckets, and how it’s used to be able to store your data, and how to know which bucket your data is in. Find out more about the different buckets in Splunk right after this.

Today, we’re going to be going through the five different buckets in Splunk, and we’re going to be talking about that. If you have a question, remember, throw it in the comment section here below. Find me on Twitter, on Instagram, and I’ll do my best to answer those here on the show. Today, I wanted to go through the different buckets and how those are used in Splunk. Before we jump in and talk about those five different buckets, let’s just get a quick definition of how Splunk works with storing our data. Think about our data coming in, to our Splunk environment. The first thing that’s going to happen is, whether we’re uploading it, or whether it’s live streaming data, it’s going to be indexed. That index is going to help us, one, be able to search it a little bit better. Splunk’s going to put a timestamp on it, and it’s going to do some other things to give us some meta data, so that we can simply search through that data a lot quicker in our Splunk environment.

The other thing it’s going to do is, it’s going to store that data so we can find it. It’s going to store those in different buckets. Think of buckets just as the Splunk file system. Just like you have a file system, think of it in a Windows environment. I’ve got directories and subdirectories. Think of it in the Splunk environment as you have different buckets. I have buckets for different portions of my data. Those are all going to be with a timestamp. Right? Right. As it indexes, that’s how Splunk decides where they’re going to be in the bucket, and also, there’s some other things you can do to decide how long data’s going to sit, and sit in each one of your buckets, but before we jump in and talk about that, let’s make sure we understand what those buckets are.

The first bucket, or really the first two buckets, are going to be your hot and your warm bucket. Your hot and your warm bucket, this is where your most recent data is going to be. Your hot and your warm bucket, they’re both going to sit in the same specific area. They’re going to be put in there so that you have your data the most current, right? This is where Splunk really puts a lot of performance characteristics around where this data should live in these hot and warm buckets, and specifically really, if you think about it, your hot and your warm bucket, your warm bucket’s going to contain some of your most recent data, but your hot bucket is going to be the one that’s riding to the new data. As you set up your policies for how long your data is going to exist in your Splunk buckets, your hot and your warm bucket, let’s say that arbitrarily you can get 10 events. It’s a little bit more complicated than that, but let’s just make it simple. Say that you can get 10 events in your buckets. Every time you get to 10 events, your hot bucket is going to become a warm bucket, and you have a brand-new bucket. Think of your hot bucket as where the newest files go, and your warm bucket is where your more recent is. It gives you a life cycle policy.

The third kind of bucket is a cold bucket. This is where our more older data, where our data is kind of aging off. This data can actually live, doesn’t have to live with a hot and warm bucket. It can go to maybe a NAS device or some kind of object store where you can actually search on it. It still needs to have some requirements for how it’s being searched, because it could be, if we’re saying that we have 10 events in each one of our warm buckets, let’s say that after a week, those 10 events age off to our cold bucket. Our cold bucket could hold from a week to maybe three months on our policy. That’s where our data is going to exist, in that cold bucket. No new data’s being written to it, and there’s not as much performance requirements just because, probably not searching on it as frequently as we are on the newer events. They’re pulling out for our dashboard, then are stored in our hot and our warm buckets.

Then, we have our fourth bucket, which is going to be our frozen bucket. Think of this as really old, frozen data, hence the frozen bucket. Data that we’re holding onto for compliance reasons or we just want to be able to go back and search on it at some point in the past, but this data is actually going off to some kind of long-term retention. The thing about it is, we want to search on it, I’ll talk about it in just a second, but this data is not searchable in its current form of a frozen bucket. There’s another process to be able to do that, and that’ll include another bucket, but think of this as where you’re aging off your data. This gives you an opportunity to get a better cost per terabyte for how you’re storing the data, and get it out of your Splunk search, so better performance on your Splunk search as well, but still being able to hold on to that data, but think of this as, this is, if we’re saying three months is what we’re going to hold in our cold bucket. Think of it being more than three months that’s going to exist in that frozen bucket.

In our last bucket, number five just talked about it, it’s a thawed bucket. Our thawed bucket is how we get that frozen data back into a searchable state. You can go through and being able to thaw that bucket out. Think of it as taking some of the compression out of it, but also putting it in a better place to be able to store it. We talked about performance, and some of the other characteristics that you need to be able to search your data. In those thawed buckets is where you can start and go from that process. It’s a full life cycle process. Go from hot, to warm, to cold, to frozen, and then when we want to see your data again, put it in that thawed bucket. That’s all we have today. I hope you enjoyed this episode, where we talked about the five different kind of buckets in Splunk. If you have any questions, make sure you put them in the comments section here below. Reach out to me on Big Data Big Questions, and I’ll do my best to answer your questions right here.

Want More Data Engineering Tips?