For a while now, I’ve been running into a series of situations where, when we’re helping a client with a data environment, the success of the project comes down to making data providers talk to each other, preferably seemlessly.
This can be any combination of things. From applications from third parties that gather and store information and have “claimed” that they provide database services (only to find out it’s simple extracts with no unique keys, etc.) to the more traditional SQL Server, MySQL, DB2 environments to all of the above plus cloud-based database resources.
These different data sources present differently – they may be feeds of information, may have different SQL capabilities, etc.
One of the things we’re finding more and more time-consuming is making these connections. Different data type treatments, different import data types (default to nvarchar() for example, when it’s numeric). Things like this, combined with different actual locations and sources for both the data and the data engines makes it very interesting to make it all talk correctly.
While I don’t know that this is a new problem (how long have we collectively been working with different data sources?) perhaps the biggest difference is that the information seems to flow differently, and the expectations are that the processing will be responsive right along with it.
Either the data flow is constant and needs to be always current, or the application supports query and re-query to pull information for the user to handle that information. Both of these mean the conversion and data cleansing needs to happen very quickly and be aware of sources, destinations and uses. It can be very challenging.
How are you addressing this? Do you just bite the bullet and do the integration work, or do you look for third party applications to play a bit nicer and provide common-format data files or do you just not work in this type of environment? Inquiring minds want to know – just shoot a comment below.
In the meantime, keep an eye on those data types and the implications of implied data conversions, lack of keys and different platforms and tools. It certainly keeps things interesting.