7 key criteria to evaluate IoT automation tools

Technology criteria



Modeling complex logic

Real life is multivariable.

The engine should support:

  • Combining multiple non-binary outcomes of functions (observations) in the rule, beyond Boolean true/false states.
  • Dealing with majority voting conditions in the rule.
  • Handling conditional executions of functions based on the outcomes of previous observations.


Modeling time

Time adds complexity.

The engine should support:

  • Dealing with the past (handling expired or soon-to-expire information).
  • Dealing with the present (combining asynchronous and synchronous information).
  • Dealing with the future (forecasting for prediction and anomaly detection).


Modeling uncertainty

Uncertainty is unavoidable.

The engine should support:

  • Dealing with noisy sensor data and missing data.
  • Dealing with unstable wireless sensors, fully dependent on battery lifespan.
  • Dealing with intermittent network connectivity or network outages.
  • Dealing with unreachable API endpoints.

Implementation criteria



The engine should be explainable, allowing users to understand why rules are fired and to identify and correct errors. The engine’s internal complexity should not come in the way of its users being able to easily test, simulate and debug that complexity. Users also require a high level of understanding and transparency into decisions with inherent risk.



The engine should be flexible enough to support both commercial and technical changes with minimum friction, such as changing customer requirements or changes in APIs. In order to account for future growth, the rule engine should be easily extendable and capable to support integration with external systems.



The engine should be operationally scalable. When deploying applications with many thousands or possibly millions of rules running in parallel, the engine should effectively manage the large volumes, by supporting templating, versioning, searchability, bulk upgrades and rules analytics.



The engine should provide a good initial framework and abstractions for distributed computing to enable easy sharding. Sharding refers to components that can be horizontally partitioned, which enables linear scaling – deploying “n” times the same component leads to “n” times improved performance.

Download the full benchmark

Complete with extensive definitions and examples for each of the seven evaluation criteria.

Most popular rules engines for IoT application development