Behind the Scenes: The Standardisation Process

POV: You're trying to integrate with Garmin, Google Fit, Oura, Wahoo and Fitbit for activities data to display your users' heart rate during a workout on your app. You go to each and every one of them and apply for an API Secret/Key. You read their docs and find out they use different OAuth Systems. You then continue reading and find out their means of sending data, structure in sending data and field names are all different. You are on the verge of giving up. Then, you found Terra

The Problem

It is no secret that the flow of data for each provider is similar especially if they have a REST API making for "easy" integration. However, even if that is true, you would often find them all using different structures (names, units, fields structure, etc) in data sending.

Let's take Google Fit and Wahoo for comparison on how they send data through their REST APIs:

Wahoo will provide each of the workout in a response in the following way (example provided by Wahoo Docs):

{
    "id": 8297,
    "ascent_accum": "450.00",
    "cadence_avg": "50.00",
    "calories_accum": "1500.00",
    "distance_accum": "24909.71",
    "duration_active_accum": "179.00",
    "duration_paused_accum": "95.25",
    "duration_total_accum": "275.00",
    "heart_rate_avg": "100.00",
    "power_bike_np_last": "150.00",
    "power_bike_tss_last": "304.90",
    "power_avg": "94.59",
    "speed_avg": "10.75",
    "work_accum": "1041480.00",
    "file": {
        "url": "https://server.com/4_Mile_Segment_.fit"
    },
    "created_at": "2018-10-23T20:43:50.000Z",
    "updated_at": "2018-10-23T20:43:50.000Z"
}

Now after receiving this payload, you would have to also download the file from the file.url field programmatically. You then must write a FIT File parser, and parse the data within (similar to a csv file) for the heart rates data.

Google Fit on the other hand, provides heart rate data in a complex JSON format:

{
  "startTimeMillis": "1651590609583",
  "endTimeMillis": "1651591457304",
  "dataset": [
    {
      "dataSourceId": "derived:com.google.activity.summary:com.google.android.gms:aggregated",
      "point": []
    },
    {
      "dataSourceId": "raw:com.google.heart_rate.bpm:nl.appyhapps.healthsync:walking heart rate ",
      "point": [
        {
          "startTimeNanos": "1651590729582000000",
          "endTimeNanos": "1651590780242000000",
          "dataTypeName": "com.google.heart_rate.bpm",
          "value": [
            {
              "fpVal": 115,
              "mapVal": []
            }
          ]
        },
        {
          "startTimeNanos": "1651590780242000000",
          "endTimeNanos": "1651590785242000000",
          "dataTypeName": "com.google.heart_rate.bpm",
          "value": [
            {
              "fpVal": 96,
              "mapVal": []
            }
          ]
        },
        ...
      ]
    },
    ...
  ]
}

Looking at this, you will have many different datasets to go through along with scanning to make sure the dataSourceId matches what you're looking for. Then you will have to parse every point list there is for the data, dealing with nanosecond mixed with millisecond timestamps along the way.

This is only two (out of the 24) of the providers and we can already see their data structures are completely different and would be tedious and completely horrific to deal with. It leaves a lot of room for bugs such as misspelling the field names, miss handling of null values, mistaking the datatype passed to you and etc. It then also leaves a lot of room for misinterpretation. For example, you know that the UNIX timestamp in the Google Fit response is given in UTC timezone. However certain providers give responses in user's timezones and provide it in a different format (such as the ISO Format you see in Wahoo's example).

There is still another caveat where every provider have their own choice of units selection. Heart rate by default has usually been passed as beats per minute or BPM, thus there usually isn't much we need to deal with here. However, providers sometimes provide units in user's preferences. Let's say a user lives in the US and another lives in France. The provider will provide the data from the user in France in the metric system: meters, kilograms, liter, etc, while the user in the US will have their data transferred in the imperial system: inch, feet, pounds, gallons and etc, making it a complete nightmare to convert.

The Standardisation Process

The above describes the main problem most developers would face integrating with many different wearable device data providers. We, Terra, deal with all these problems for you (did I mention: for over 24 providers?) and standardise it all into one format: The Terra Format. Wether you communicate with us through either REST API or SDK's, you would never have to deal with all this hassle ever again.

We will take the data from the providers and parse it into our data models, standardising, converting and filling in all the fields as correctly as possible. We also explicitly name each of our fields with their units so when you're developing, you would never need to dig through our docs or responses to find this out (like we needed to for some providers).