If you thought “Big Data” was already quite unmanageable, IEEE predicts a 1500% (x15) growth in data by 2015. That is 3 years from now.
On a similar scale, IEEE also suggests that terabit networks should be implemented soon to cater for demand in network traffic by 2015. This is up by x40-1000 times from today’s gigabit networks.
This probably also suggests that demand for data processing and delivery will need to increase by a similar scale. To some 10-40 times.
What products and skills will power the delivery of services for “Humungous Data”?
- New Data systems – like GFS, BigTables, Hadoop, Hive, MapReduce
- New Data patterns – No-SQL
- Cloud computing – A must for elastic computing vs BYO data centres
- Open data systems skills – unless you plan to pay for expensive database licenses.
- Web Services – to tie it all together
- Agile Architecture – often under-rated, but is increasingly important to focus corporate development.
- Agile Security – also under-rated, but is increasingly important.
With corporations already struggling to manage data growth and demand, will this mean a growth of x15 in data staffing, or will a data specialist have to be x15 times more productive. I believe its a combination of both. New tools will make the data professional more effective. At the same time because of the lack of training and skills transfer, there will always be a need for the human bridge.
The future is indeed exciting.
