Metering Design Principles
Core Design Principles of the Amberflo Cloud Metering Service Platform
A core design principle of the Cloud Metering Service Platform is guaranteed accuracy.
To achieve this, the system must be designed from the ground up with three key attributes:
- Accuracy
- Idempotency
- Data Deduplication
1. Accuracy
The data must be fully accurate to allow the metering system to serve as the single source of truth for usage and consumption. Other systems can rely on the completeness and correctness of Amberflo’s metering data for critical functions such as invoicing, billing, and reporting.
2. Idempotency
Each record must be processed once and only once.
- It must not be skipped.
- It must not be processed more than once.
- It must be processed exactly one time, regardless of any system failures or retries.
Example: If a valid meter event is ingested and the system fails during processing, Amberflo ensures that the event is still processed exactly once. This eliminates the risk of dropped events or duplicate billing. The idempotency guarantee is built into the Amberflo Cloud Metering Service by default.
3. Data deduplication
Data deduplication is a technique for eliminating duplicate copies of repeated data. This can be a common occurrence, depending on the source generating the meters, and it is challenging to guarantee deduplication if the processing system (Cloud Metering Service Platform) itself is a highly-distributed stateless system. In a distributed system, any part of the system or node can fail at any time, and if it fails mid-stream as data is undergoing transformation or processing, it makes it very difficult to "guarantee" data de-duplication.
In cloud metering, it is common for the source (client generating the meters) to send duplicate meters. Because the cloud metering service serves as the system of record and the single source of truth for usage and consumption data, duplicate records must be deduped.
Data deduplication is a core platform tenet to deliver accurate usage data to downstream systems such as pricing, billing, and others.
Amberflo Cloud Metering Service Platform provides you with an out-of-the-box data deduplication guarantee. Here are some different ways how you can enforce data deduplication using Amberflo:
The Amberflo platform will not store duplicate data for the same meter record. If you call the ingest API with the same record, the meter repository will only hold the first record and discard any subsequent records. The deduplication key is based on all the meter attributes. That means if one of the meter record attributes is different (either the record time or any dimension value) we will consider it as a different record.
1. Unique ID: You can enforce deduplication by adding a unique identifier to each meter event using a dimension:
dimensions_with_unique_id.append({'Name': 'unique_id', 'Value': str(uuid4())})
metering.meter(options.tenant, options.meter_name,int(options.meter_value), dimensions=dimensions_with_unique_id)2. Unique timestamp: if we want to make sure we use one meter for the current hour/minute/second, we can set the timestamp of the request
metering.meter(options.tenant, options.meter_name,int(options.meter_value), dimensions=dimensions,timestamp=str(int(round(time.time() * 1000))))Configurable Deduplication Logic
Amberflo allows you to customize the logic used to identify duplicates. While the system defaults to using a generated unique ID, you can configure it to use any dimension key that aligns with your data model.
If an ingested event has a duplicate identifier based on the configured logic, the event will be rejected.
Real-World Example: Avoiding Duplicate Charges in NLP Pipelines
Problem:
A vendor uses Amberflo to meter the processing of text blocks for customer support cases using a text-blocks-processedmeter. Occasionally, the vendor must reprocess historical data due to onboarding workflows or failures in their machine learning pipeline. Customers should not be billed multiple times for the same block.
Solution:
Amberflo implemented configurable deduplication logic using the block-id as the uniqueness key. Before processing, the pipeline checks whether the block-id has already been ingested. If it has, the event is discarded to prevent duplicate billing.