Why do we need a census?

Last week I attended a “Beyond 2011” Census event organised by the Prof. Dave Martin and the Office for National Statistics (ONS). The attendees came from central and local government, private companies that utlise census data, and a few universities. The majority there (based on an approximate straw poll) believed that there would not be a census “in its current form” in 2021. This is because census data collection is a very costly exercise and its results are becoming more out of date more quickly as the pace of societal change (through migration, for example) increases.
So, we need to find an alternative but from where? Census data collection has been taking place for centuries and has, until relatively recently,  been our only source of “Big Data” concerning the population characteristics of the UK. Why then, when we have finally achieved the computational and analytical capabilities to efficiently analyse it are we prepared to throw it all away? The reason relates to the fact that we have hundreds of big data sources available now and, if used properly,  we can potentially generate the same levels of insight (and more!) as those provided by a traditional census form.

We live in an era of "Big Data" as Techcrunch (click image) explains

For example the 2011 Census only accurately recorded where people live, not where they work. What good is this for emergency planning in central London? Why not use Oyster Card trip data, or mobile phone usage instead to give daily updates on population movement? Another issue is the fact that we have a more mobile population and we miss a lot of change by only taking decennial snapshots. Other government datasets are updated more often, such as the NHS Patient Register or the Electoral Roll, and can provide an indication of where we are living this year, rather than where we lived 5-10 years ago (I moved house 4 times between censuses). We also, and perhaps controversially, hand over loads of personal data every-time we use a store-card or credit card, log into Facebook, use our mobile phone or surf the web. This has contributed, perhaps for the first time, to many private companies having a much better idea about aspects of the population of the UK than the government. The question is do we want companies to share it for the greater good (or evil depending on what you think of the “big brother” state), or should we let them keep the data and have the government spend more to source it itself? We also have to be sure to count those who don’t feature on private company databases (put crudely, often because they aren’t worth anything to that company) and it is these groups, often the most vulnerable in society, that we are most likely to miss with a non-census solution.
I think there is more than enough data to go around without having to fill in lengthy census forms, the issue is we haven’t worked out how to join it all together yet. Once we solve that problem we then need to work out who we have missed and that is much harder to do without a compulsory census!
If you want to contribute to this debate please fill in the consultation documents.