Scope of this Document
This document outlines a new enhancement to the Shoplogix Analytics architecture which mitigates issues relating to partial / incomplete data in the Analytics Portal.
Details About the Partial Build Failures
Background: During the pre-stages before a cube build in Sisesne, some core data queries are producing incomplete information, which is later used to build a cube.
Issue: Sometimes individual query batches to a Saas server will fail or time-out, which results in storing incomplete data into its intermediate database (i.e. missing data for some machines). Also on rare occasions, a server may be temporarily unavailable which can cause an entire server to be skipped. This partial data is ‘staged’ for an upcoming cube build. Later, when the cube builds, it will build using this partial data. This partial data will then be visible to users in their Analytics dashboards.
Impact: Primarily the live-data window (within 45 to 60 days), for random groups of machines and servers. This missing data is temporary until the next build cycle.
Enhancement: Skipping Partial Cube Builds
In order to protect against partial data being built into a cube, a new mechanism has been implemented which will detect and skip builds using partial data. Partial data refers to cases where either:
One or more servers that should be included are missing from the data
Less than 100% of the machines on a given server return data when queried
When partial data is detected in the intermediate database, the ‘trigger’ to build the next cube will be disabled, which effectively will skip that build.
Result / Side Effects of Skipping Partial Builds
When a build is skipped due to detected partial data, it means that the previous cube build will remain as the source for analytics dashboards. For example, let’s say cubes build every 30 minutes, and the last successful build was at 10:00 AM. If a the 10:30 build cycle has partial data detected, it will be skipped, and the 10:00 AM cube build will remain in place until it is later replaced by the next build cycle at 11:00AM.
Although unlikely, if more than one partial build occurs in a row, the ‘skipping’ will continue, and the period between cube builds will be longer than the typical expected cycle.
Appendix: Analytics Architecture Overview
