Administration, Azure, Azure SQL Database, Editorials, Encryption/Data Security, Ethics, Security

Data Architecture Challenge – “Active Archives”

Perhaps one of the biggest challenges that we’re all working through right now is working with the volumes of data on hand, and maintaining it in a usable way.

We’ve been working with a couple of different clients to create ways to either use some of Azure’s Elastic Database approach, or move different information bits to different systems that would be pulled together as needed.   We’re faced with a number of options – each of course has its pros and cons, but it’s interesting to work through what it means for applications, development and analysis and reporting later on, to choose the various options.

There are prior posts here (even recently) talking about how the legislative and cultural changes that are happening will impact information stored.  It’s funny to have a conversation with a “higher up” about privacy – you start with outrage and hearing that they think things are being abused and want to control it and all of that.  But then when the discussion comes around to their own storing of information, or the vision of what that information does and must mean for the company, it’s quickly obvious that it’s “store it all!” and figure out what to do with it all later.

So, we’ve been starting to really re-think how we work with people in this type of situation.  We’re helping them work through things in an evolving way.

Transparency (policies, notice, response)
This first one goes to legal and privacy policies and things that those that give information need to know.  How will it be used, what is gathered, all that stuff that we all see in the privacy and data use policies, but, frankly, ignore.  It needs to be defined, clear and concise.  It also needs to be reality-aware.  So many times I’ve seen a blanket statement offered up as the policy, only to find out that this or that element, or cookies, or whatever other element really is being used.  Having this transparency is the only way using that information later, will work.  If you start offering customers a better buying experience, sooner or later, they’ll figure out that you’re keeping information elements to make that happen.  Be clear about it, no surprises.

Data Plan (storage, security, etc.)
To me, the data plan has always been about delivering on the Transparency and Application Plans.  Ironically it’s also a bit of a circular reference on the Application Plan because the application plan will need the data elements set up. This needs to address the storage elements (including archive storage, storage automation, burstable use, etc.) and security of that information – and don’t forget the different phases of security (acquisition, transit, storage, analysis, archive, removal, etc.)  How is that information going to be handled.  It’s like a bank.  Data is the cash.  You better know how to manage it, store it, protect it, etc. in order for people to trust you with their cash.

Application Plan (development, retro-fitting existing applications)
Just like it seems, the Application Plan will talk about how information will be accessed, what the expectations are for data use and how you’ll support that with the development work that is needed.  “Development” doesn’t just mean that C# application, it includes Excel, PowerBI and all of the other tools that will be touching the data.  If it’s for an existing company, you have an already existing challenge; you need to figure out how existing applications will be fitted to use the new data systems.  You’ll also have to determine if changes are needed, and how new systems will be talking to older systems and still honoring your commitments elsewhere in this list.

Implementation Plan
This is the “getting it all done” setup.  How will you go about this?  I’ve been working with a company that being transparent is an incredibly difficult and winding road for them.  Not because they’re hiding anything, but the fact that they’ve not had to worry or think about it to-date has been very convenient.  Other companies may have zero archive or security plan.  It’s a matter of figuring out how you’ll be applying the things above, and what you need to do to determine “success” in those efforts.

It can be a bit of a shock to the system, and it will evolve along the way, right along with society as we go through all of this, but it’s important to take a step back and add some structure to the overall picture… for your own sanity and that of your stakeholders.