When data is neither big nor small, what is it?

This blog post was written using the Recorder app on a Google Pixel 3a phone. It transcribes speech so there may be a few small errors but I've corrected what I could see. Enjoy!

I wanted to talk about concept for a while now that tries to align big data with what people are really doing in the real world which is often called small data, which is defined as the amount of data that's small enough for human comprehension. Now, obviously lots of organizations have been talking about big data for a long time and there are lots and lots of services to help with that and then this this small data concept has been around for a little while trying to kind of tackle what people can actually manage.

But what the problem is that we need something in the middle and lots of organizations can do big data some still do small data, but there are a huge bunch of organizations in the middle that can't do big data. And is big data the right concept for them?

It's used to describe pretty much all data and all analytics today. So we need something a bit different. I've been thinking about that for a while now and with my partner being Swedish, I thought about a word that they use in Sweden. It's a word that doesn't have a direct translation.

It means just enough or about right and it's a concept that's very prevalent in Swedish society in that everything is just about okay, it's fine. Doesn't have to be too much or too little. People don't need to be too rich or too poor. You don't need to eat too much or too little.

You have just enough. And that word is Lagom. Pronounced L-A-R-G-O-M. So I've been looking into the concept of should we describe data that we're analyzing but it's too big for comprehension, but isn't the size of you know, the feeds coming in from the larger ray telescopes as something called Lagom data something in middle?

It's just the right amount of data. You could also call it Goldilocks data. So looking into a quick Google search for Lagom and data there's a couple of concepts similar out there already and there's a framework for API microservices called Lagom framework, which is about managing data persistence.

That's all APIs but not necessarily about the size and shape of data but there's an organization in Brazil called Lagom data. And they essentially are a consultancy who are trying to help organizations with the data issues they have. So it's a concept that isn't necessarily new but also isn't huge out there right now.

I think it's something that maybe we could be talking about a bit more. Certainly internationally it's something that could be quite interesting but it's concept that hopefully helps us describe what we do as data people a bit better. You know, we're not always looking to process and vast billions of rows of data.

We are sometimes looking at those data sets of a thousand or two thousand rows that can really make a difference to what a business can do and often it's smaller organizations that can really benefit from the power of analytics but they don't have the volumes that others would have and maybe is that concept of Lagom data that could help them.

Additionally under GDPR, we all need to be very careful about how much data we are collecting and it needs to have a purpose and that has seen the volumes of data that organizations collect drop in some cases and again, you know are they true big data organizations anymore are they collecting everything they can with the hope that they'll find use.

For it or either being much more targeted and making sure they're abiding by the letter of the law and again the concept of Lagom data just the right amount of data could be useful for describing what they're doing today, so it's concept that could be worth exploring a bit more be interested to hear your thoughts on it.