Getting the Big Picture – Part 3
Today I would like to talk a little bit about monitoring for distributed processes, and how that changes the big picture when you get complaints about performance.
When I am talking about distributed processes I am talking about sharded data or logic; sometimes both. Sharding your data in an SQL or NoSQL environment works essentially the same from a consumer perspective. A central broker of some sort is responsible for locating the shard where your data is located in order to execute you query. If your query is broad enough in scope, or the sharding is not based on an efficient grouping for the query, there can be a lot of extra work required to fulfill the request.
The number of hops to different servers is usually a positive thing in sharding. This is what provides the parallel searching or processing capabilities resulting in faster performance. What is not helpful, however, is when the shareded queries have to once again hop to other servers for portions of their work, resulting in more work than if the data was not sharded to begin with.
If you are using a sharding technique, it may make sense to instrument your application for sharding hop events, defined as when a shard is not able to completely fulfill the requested work without having to request data or processing from another shard. Perhaps your software already has some sort of monitoring of this nature already built in.
This concept works just as well on large data mining platforms from vendors such as Micrsoft, IBM, Oracle, etc. In these large environments a lot of data is replicated across multiple shards with the intention of having only a single source for resolving a query on any sharded instance. Perhaps you have not replicated enough data; or have replicated too much resulting in unnecessary update overhead and added expense.
What kinds of tools are you using to monitor sharded data? Please leave a comment below with your experience, or drop an Email to btaylor@sswug.org.
Cheers,
Ben
$$SWYNK$$
Featured Article(s)
Tips for using SQL Server 2012 file and filegroups
In this article, you can find some tips to use file and filegroups in SQL Server 2012.
Featured White Paper(s)
Encryption & Key Management for Microsoft SQL Server 2008/2012
Written by Townsend Security
Simplify encryption and key management on … (read more)