CleanShrew1 week ago
I'm building a feature for my application that requests historical data when a new device is connected through Terra. I've been testing the Terra endpoint to request historical data but I've noticed I never get the amount of data I'm requesting. On the best scenario I get up to 4 weeks of data. Subsequent calls to the endpoint return no data even when they have a 200 status.
I'm testing with my own data and I know there is more available. I can request data as old as 6 months or a year and it works but the return doesn't cover the full date range. Only when I request less than 28 days with to_webhook parameter set to false, I get the 429 status response indicating there is a quota. Any other type of request sending data to the webhook results on a 200 status but no data.
I haven't found any documentation about this quota so I'm unable to build my backfill feature around that. I also asked through support but haven't got any help. Is there any information regarding a quota when requesting historical data?
Vanessaadmin1 day ago
Hi there!
There might be different reasons you are not seeing the entire range of data you had requested:
Provider-Specific Historical Data Limits Some providers impose strict limits on historical data retrieval :
- Garmin: 5 years into the past from the current point in time
- Polar: no historical data retrieval permitted
- Health Connect: 30 days into the past from the moment the user granted permission
Large Request Handling For time ranges longer than 28 days, Terra defaults to sending data asynchronously, even if to_webhook is set to false . This explains why you're getting 200 status responses but no immediate data for longer ranges. When making large requests, Terra sends data in chunks of :
- at most 10MB
- at most 10 objects in the data array
All chunks will contain the same terra-reference header value!
**Asynchronous Data Processing ** Following a large request, you'll receive a large-request-processing event indicating Terra has acknowledged the request and is processing data retrieval. Once complete, Terra sends a large-request-sending payload indicating the number of payloads to expect.
Rate Limiting Considerations Some providers have strict rate limits which will result in a longer period before the request is fulfilled. This could explain why subsequent calls return no data - the provider may be rate limiting your requests.
Recommended Approach To handle historical data requests properly:
- Use to_webhook=true for requests longer than 28 days
- Track the terra-reference header value to match incoming webhook events
- Wait for all expected chunks before considering the request complete
Hope this helps - let me know if you have any other questions!