To make sense of the enormous quantities of data generated by a city, it is imperative that information is stored properly for subsequent analysis and dissemination. When deciding what kind of storage solutions and process tools to use, there are several factors that local authorities should take into consideration.
The starting point is always to consider the policy issues and questions to be addressed. These directly influence the types of data to be collected and their likely storage requirements. Policymakers should also clearly identify the legalities around what information can be stored, how it can be retained and where the trade-off between data retention and destruction should be.
Frequency of data updates
It is also essential to strike a balance between the frequency of data updates and the consumption of resources. Real-time data processing plays a vital role in many smart city applications, but is generally very resource-intensive. Choosing the appropriate frequency of data collection, therefore, involves a clear trade-off: consistency and accuracy of data collection versus storage capacity and cost.
Centralised versus distributed storage
Once the desired output and frequency are clear, cities often have to weigh up the trade-offs between physical and centralised storage solutions or those in the cloud. Physical storage infrastructure is likely to be more suitable to smaller databases that are confidential and highly sensitive in nature. The advantage may be on-hand security, control and customisation, but the disadvantages are typically low scalability, high maintenance costs, and the technical responsibilities of managing such a system.
An alternative is cloud storage, where unlike traditional centralised servers, which can only be accessed on site, cloud systems allow for distributed storage of data supported by a dispersed network of computers across multiple locations. Cloud systems have the key advantage of being easily scalable and flexible, as new server systems can be quickly added as and when an organisation needs. Such systems also make it easier to store data securely, as users are no longer dependent on the functioning of a single server, which, as a singular system, may be more vulnerable to cyber attacks or physical damage.
Lastly, because cities are effectively leasing server space as opposed to investing and constructing the servers themselves, cloud storage can be a much cheaper alternative to traditional methods.
There are three broad categories of cloud systems:
Cities use the public cloud by renting a slice of distributed data-centre infrastructure from a service provider. The advantage is its scalability (unlimited on-demand cloud resources), lower costs (no need to spend on equipment and IT maintenance) and reliability (as the services are distributed to different data centres). Yet, control over data security may be a concern.
A private cloud is a dedicated cloud infrastructure that can be managed by the city itself or by hiring a third-party service, hosted in the city data centre or off premises. The advantage is its security, control and customisation, as well as the flexibility of being able to move non-sensitive data to a public cloud when needed. The drawback is its higher cost and the responsibility involved in operating and maintaining the data centre.
A hybrid cloud is a mixture of on-premises infrastructure, as well as private and public cloud services. This mixed computing and storage environment allows data and applications to be shared between the various platforms, from private physical storage to public cloud.
Data processing follows a structural cycle starting with the input stage – which is where collected data are converted into machine-readable formats.[iv] The next stage of processing is where raw data in the form of images, videos, text, numbers and other alternatives will be classified and analysed to yield meaningful results. This can happen either through a local server or in the cloud, depending on the authority’s needs, and once transformed this makes up the final stage of output presented to the end-user or stored for future use.
The idea of processing raw data near its point of origin (that is, the sensor) before sending it to be stored is an important consideration for municipal governments and one behind recent innovations in Edge computing.[v],[vi]The advantage is that only the processed data and essential information will be transferred over the network, requiring less bandwidth and leading to lower battery consumption by sensor devices.
Ultimately, the most appropriate storage solution for a city will depend on its budget, scale, data sensitivity and desired objectives, among other factors. Cities may look to use proprietary or national, centralised server systems where available, or cooperate with solutions providers to form their own cloud. The main advantage of cloud is its scalability and flexibility, as well as its potentially lower cost, but cities should carefully consider their capacity to uptake such systems when designing their approaches to storage.
Resource implications and key requirements
The selection of storage solutions is largely subject to city government budget. Local authorities can either rent the services or build their own storage and process platforms – with the costs of building large, independent service centres likely to be beyond the realm of many smaller municipalities. Cities with smaller budgets may wish to establish a data retention policy, which is essentially a directive for data to be deleted after a certain period, so that data are not kept beyond their functional lifespan and space is consistently freed up for new uses. The trade-off here is that shorter retention times, and ultimately fewer data, may lead to fewer historical references and insights.
For those looking to invest in the cloud, before contracting with service providers, there are a few things to think through and assess.[vii]
- Scalability – ensure the chosen platform has the capacity to scale up when needed.
- Security – check the security of data storage, processes and user accessibility.
- Data ownership – most importantly, ensure the city government is the data owner, not the service provider.
Many cities have benefited from cloud services. Manchester City Council in the United Kingdom,[viii] for instance, moved more than 900 application workloads to the Nutanix Enterprise Cloud over the course of four months. It saved the city £285,000 a year by significantly reducing power, cooling and rack occupancy, which decreased 90 per cent. Lenoir City Utilities Board[ix] in Tennessee, United States, cut its operating expenses by two-thirds by migrating all data and applications to the cloud after a 20-fold performance increase in the production of structured query language (SQL) databases and virtual servers.
Potential private-sector participation
As with data collection and transmission, governments are likely to involve the private sector with data storage and process infrastructure to varying degrees. The private sector can support the supply of vital infrastructure through traditional procurement models, and may even take on greater levels of risk sharing – for instance, through contracts for IT operation and modernisation, implementation of cloud-based solutions, and server efficiency improvements, among others.
Large-scale infrastructure such as data centres are also likely to be well placed for typical PPP models, given their high construction and operation risks, and the requirements for heavy investments in space, power, cooling, connectivity and security.[x] In this way, governments could look to lease space for their IT equipment rather than build their own facilities, and this applies equally to physical data centres as well as cloud-based operations. One thing to consider here, however, is that private-sector operators may have greater proximity and access to sensitive public data. For this reason, it is particularly critical that governments work with trusted partners who place a high degree of emphasis on the security of their systems against internal and external breaches.
Obstacles to implementation and potential solutions
There are clear trade-offs between different data storage and processing technologies, and the likely costs incurred by municipalities. Cities with smaller budgets might gradually start with local servers and public cloud capabilities, while cities with more significant budgets might adopt a hybrid approach for its scalability and versatility. Large quantities of data can be retained over time using cloud computing solutions, so this makes it an important facilitator of long-term policy analysis and decision-making.
Obstacles can often occur when a government first adopts cloud solutions. Some staff may be hesitant about the security and storage capacity of a non-physical system, while others may see the administrative processes of uploading data to the cloud as additional workload or potentially problematic if seen by other departments. Behavioural-change campaigns, skills training and well-structured institutional frameworks that clearly identify what and how certain data can be shared will play a key role in fostering such processes.
[i] Deloitte (2020) “Smart Cities and the journey to the “Cloud””. Available at: https://www2.deloitte.com/content/dam/Deloitte/us/Documents/about-deloitte/us-about-deloitte-smart-cities-journey-cloud.pdf
[ii] Nichols, M.R. (n.d.) “What Type Of Data Storage Do Smart Cities Need?” Available at: https://www.smartdatacollective.com/what-type-of-data-storage-do-smart-cities-need/ (Accessed: May 2021)
[iii] NetApp (n.d.) “What is hybrid cloud?” Available at: https://www.netapp.com/hybrid-cloud/what-is-hybrid-cloud/#:~:text=Hybrid%20cloud%20refers%20to%20a,orchestration%20among%20the%20various%20platforms. (Accessed: May 2021)
[iv] FUZON (n.d.) “How IoT Works – Data Processing in Internet of Things (IoT)” Available at: https://www.fuzon.io/insight/how-data-processing-works-in-iot/#:~:text=All%20solutions%20in%20IoT%20typically,or%20further%20dissemination%20of%20information (Accessed: May 2021)
[v] Junnila, A. (2018) “How IoT Works – Part 3: Data Processing”. Available at: https://trackinno.com/2018/06/04/how-iot-works-part-2-connectivity/
[vi] Nichols, M.R. (n.d.) “What Type Of Data Storage Do Smart Cities Need?” Available at: https://www.smartdatacollective.com/what-type-of-data-storage-do-smart-cities-need/ (Accessed: May 2021)
[vii] NetApp (n.d.) “What is hybrid cloud?” Available at: https://www.netapp.com/hybrid-cloud/what-is-hybrid-cloud/#:~:text=Hybrid%20cloud%20refers%20to%20a,orchestration%20among%20the%20various%20platforms. (Accessed: May 2021)
[viii] Nutanix (2019) “Building the City of Tomorrow;
Serving the City of Today”. Available at: https://www.nutanix.com/content/dam/nutanix/resources/case-studies/cs-manchester-city-council.pdf
[ix] Nutanix (2020) “Lenoir City Utilities Board Upgrades to Nutanix Cloud Platform”. Available at: https://www.nutanix.cn/content/dam/nutanix/resources/case-studies/cs-lenoir-city-utilities-board.pdf