.svg)
This is a guest blog post by Nahi Ojeil, Vice President of Engineering at CaptivateIQ. Before CaptivateIQ, Nahi held engineering leadership positions at Dropbox and Commure. In this blog post, Nahi reveals how CaptivateIQ’s powerful and innovative technology works, along with the key features helping companies maximize the value of their commission models.
The market for incentive compensation management solutions has been in existence for about 25 years yet to date no company has been able to establish itself as a clear leader in a manner that is evident in other spaces.
The reason for this is incentive compensation is inherently a difficult problem to solve. On the one hand, customers require infinite flexibility to handle any incentive plan coupled with any number of data sources. On the other hand, customers want a simple and easy-to-use solution to guide them through standard processes. And because incentive compensation is inherently a data-rich exercise, all of this needs to be highly scalable and performant so that last-minute changes or recalculations can be achieved before the stringent payroll deadline.
We believe that CaptivateIQ has the foundations to become the undisputed leader in the space with a unique balance of flexibility, process control, and scalability.
At the heart of CaptivateIQ is a powerful and innovative technology that sits behind a familiar spreadsheet-like user interface (UI) and a robust and intuitive formula library. It is a very hard problem to take something as performant and scalable as our modeling platform and expose it as something as simple as a spreadsheet, but don’t let that fool you when it comes to what truly powers this “spreadsheet in the cloud.”
In this post, we provide a breakdown of the architecture of our platform.
5 key components of our compensation technology architecture
The major components of our product architecture consist of:
- Application Layer. This represents the cohesive experience where all the features come together for various users of our product.
- Features. This represents the rich functionality that we provide in our applications to perform various workflows and actions. Some examples include processing commissions, submitting and responding to inquiries, performing scenario planning, or building reports and dashboards to drive insights. This also includes the no-code UI that allows easy manipulations of the data model through spreadsheet-like formulas and database-like transformations.
- Data Management. This represents the powerful engine that allows users to apply logic and math for manipulating data and building models. It is both a highly performant and scalable calculation engine that offers a blend of real-time calculations to easily view changes to logic as well as a batch processing engine for cyclical calculations.
- Data Sync & Storage. This represents the inputs into the model and features. They need to be both automated and scalable to support the vast amounts of data that process as part of running the model.
- Security & Compliance. This represents the framework we use across our entire technology stack to provide in-depth defense via authentication, authorization, and accounting for all actions in our product and infrastructure. More details about all the other security measures are listed in CaptivateIQ’s Trust Report.
Let’s dive into the Data Sync and Storage layer!
In order to support the scale and magnitude of the data we have to process, we must have a scalable storage layer, as well as an integration engine that can automatically pull in all the data that is needed in near-real time.
Starting with the data integration engine, we have developed native connectors and APIs to allow us to pull in (and push out) data into and out of our product. This engine integrates seamlessly with the calculation engine and all other parts of the product (SCIM provisioning, for example) to avoid having to perform additional steps to use the data connected to our product.
For most integrations, this can be done fully self-serve, and for more complex use cases, we also expose our APIs (not to mention other simple ways to bring in data, such as CSV uploads).
All the data in CaptivateIQ ends up persisted in our scalable storage system. Our storage and data models are optimized for storing the different types of data we care about:
- Data records that represent transactions and facts that will be crunched in our model.
- Attribute data that need to be utilized within any model, which could change over time and thus need to be effectively dated, such as quota.
- Model metadata to represent the model itself.
- Other product metadata to power the product features.
Next, let’s talk about the Data Management layer!
At the heart of the model is our real-time calculation engine. Our calculation engine operates on data in 2D table format (rows and columns). It provides formulas that can operate on any data across any tables in the model, as well as transformations to generate new tables by performing operations across any number of tables in the model.
Even though this part sounds simple, it is one of the most complex pieces of technology we have built. The performance, correctness, and scale requirements are paramount — and performing the type of calculations required for many of the models built in CaptivateIQ in real time requires a significantly optimized engine to power it all.
So how do we do it? We won’t spill too many beans here but can share a few key innovations that allow us to unlock real-time modeling for even the largest enterprise organizations:
- Calculation graph clustering allows the engine to process millions of computations in seconds.
- Our smart caching system intelligently calculates only relevant cells, drastically reducing calculation time and allowing immediate feedback to compensation model changes.
- Optimized data access patterns accelerate overall performance.
This calculation engine is decoupled from the UX and model building that happens on top of it. This decoupling allows us to develop a user experience that is very familiar and simple yet can perform complex calculations at scale.
It has also allowed us to optimize the underlying engine independently without worrying about the impact on the modeling experience, resulting in rapid iteration under the hood, and providing numerous improvements over time without disrupting our customers!
This is too much of an oversimplification of the calculation engine that drives our modeling and calculation capabilities, so stay tuned for a future deep dive that will talk more about the internals of our calculation engine.
And finally, the Features that power sales compensation models
Over time, a number of features have been developed to maximize the value of building commission models in CaptivateIQ. As we introduce new features and focus on new opportunities, we are able to leverage the data and models we have built to power these features. Let’s dive into a few of them.
- No-code modeling. On top of the real-time calculation engine is our no-code modeling UI layer. This spreadsheet-like familiar user experience allowed us to create a bridge between what most customers are used to (spreadsheets) and the incredible power and flexibility our calculation engine provides.
- Payout processing. This is the primary use case for building a commission plan. We want to figure out how much each person should be paid in commissions, as well as have the traceability of those numbers to support transparency. Because of the power of our real-time calculation engine, payouts can be processed in real time (versus batched or scheduled to run on old versions of the data). This means the model could be changed and payouts could be run immediately after, resulting in much higher turnaround times. Payout processing is also very fast as opposed to having to run for hours overnight. This also means that payees can see a real-time preview of their payouts at any point. And with the recent launch of our what-if tool, they can even model different scenarios using the same model and plan logic in real time!
- Workflows. Almost always, the workflows surrounding commission payouts are complex. From approval workflows to addressing inquiries, it is important that our product offers the orchestration necessary to support the business processes that create transparency and confidence in what payees are getting paid. All while helping to ensure the right level of checks and balances and audit-ability. These collaborative workflows leverage parts of the model and attributes (organizational hierarchies, visibility permissions, etc.) as well as our data store to create a seamless experience between the admin and the payee.
- Reports and insights. It is very important to provide the right insights into a commission plan to various stakeholders. This could be a historical view of payouts and attainment, leaderboards, or more complex financial and revenue dashboards. These workflows are built to leverage the power of our model and the richness of our data and provide many ways to share and visualize the data in question while providing visibility controls.
Thanks for coming along!
It goes without saying that we have been lucky to have built our technology in the era of cloud first, leveraging the tools provided by a cloud provider to easily scale horizontally, to produce robust and scalable products. This means our architecture has been designed to be scalable from day one, compared to older technology that was originally designed to be deployed on-premise and has subsequently pivoted to the cloud. For these platforms, the technology debt is often too great to fully realize the power of the cloud, even after migrating a desktop-based architecture to the cloud.
We would also emphasize the importance of security and compliance throughout our architecture. Across authentication, authorization, and creating audit trails that we can expose to our customers, we layer in multiple lines of defense to help safeguard against malicious agents hunting for sensitive data like those which we have been entrusted to protect on behalf of our customers.
We hope this post provides a little more insight into what is behind the CaptivateIQ product, especially the technology we have built and optimized over the years to deliver a very powerful yet simple-to-use product.
It is that combination of simplicity, flexibility, and scalability that we believe is very hard to replicate — it makes our product unique and gives us the opportunity to become the undisputed leader in compensation management. More to come on our technology in the near future!
If you have read this far, I think you will find my other blog post about the key pillars of a great engineering team interesting. For those who are inspired by our product and want to help us revolutionize compensation management — we are hiring!
