What is Big Data?

When talking about ‘big data’ analysts will generally be referring to three main things. With a dataset or table, it can be ‘big’ in two ways, the number of rows and number of fields.

An example of this is in a UK dataset, you may have 60m individual rows; one for each person. However, for each individual, they could have a list of field descriptors such as age, gender and location; this can quickly get large too.

Typically you can use standard tools such as excel to work with datasets with less than 1m records and 200 rows. Once you exceed this you may want some professional support.

The third way is the number of datasets that are considered, the commonality in the term ‘big data’ generally means this. An example of this is building insight from lots of datasets (some of which will have big tables) increasing the size of the data. Another example for this is geo-fencing results vs actual footfall and POS spend.

‘Big data’ from the market can be collected from almost any topic: credit card spends, shop transactions, demographics, life stage, insurance, social media and phone data; as just a few examples.

“The best source of data for most companies is their own”

However, each data collection point has its own rules on what it can share, generally through the more detail the more cost. Depending what you want to achieve you will need different access and most likely there will be charges and costs occurring from this.

It is, however, best to remember the best source of data for most companies is their own. For example, sales reports, stock reports, website activity and CRM activity. This data has multiple benefits as it will be free and can help with strategy and efficiency.

However if internal data isn’t enough for you, here are some good sources of free data:

1. Twitter: All Twitter data is free, there are lots of records, but not too many fields.
2. Office for National Statistics: this is good for more general consumer trends.
3. AdWords: This can give you an idea of the number of people that have searched for a specific term in a given period. However, in some countries like China google is blocked.
4. Customer profiling: if you have a list of consumers or businesses there are a few places where you can get free profiling.