Attribute Mapping and Dynamic Composition of Requests in Go

Introduction

A while back, I encountered a problem while integrating with multiple external user-provisioning APIs. Each service had different fields, often overlapping, but not always. Some fields could be present in our system, others couldn’t. The challenge was how to let the user define which fields to include in each request, including “skipping” specific fields.

In a static language like Go, this can be tricky because we typically rely on structs with json:"someField,omitempty" tags. That omitempty is great for skipping empty fields at marshaling time, but what if we really needed to differentiate between “no data” and “skip this field entirely”? Or what if we want to pass an empty string that’s still significant?

In this post, I’ll walk through a few strategies for attribute mapping and dynamic request composition in Go. I’ll also show an example that came out of my own attempts to solve this in a real-world application.

The Problem Context

The primary scenario involves synchronizing user data from our system to external APIs, such as SCIM or RESTful endpoints. We want the flexibility to determine which fields we send downstream at runtime — no hardcoding.

Why bother? Because external systems might let you patch user attributes like timezone or department, but perhaps your internal system doesn’t track those yet. Still, your customers might want to specify (or skip) that field.

So, we need a flexible mechanism:

1. Define which fields exist in the external service (for example, email, name, surname, timezone, etc.).
2. Map each field to its source (maybe it’s from the user object, maybe it’s a fixed value, or maybe we skip it altogether).
3. Build a request object dynamically so that any fields marked “skip” simply don’t appear.

Approaches to Skipping Fields

Let’s consider three approaches:

1. Construct a map[string]interface{} for the request
   Instead of working with typed structs, you directly build a map of fields to values. Skipped fields just don’t get inserted into the map.
2. Marshal a struct to JSON, then unmarshal into a map and delete fields
   If you still want the convenience of a struct for some reason (e.g., built-in validations, default values), you can marshal to JSON, then convert to a map, and manually remove any fields you don’t want before sending it off.
3. Use reflection
   You could iterate through your struct’s fields, read the “should skip or not” condition from some config, and build a map from it. Reflection in Go can be a bit verbose, but it gives you detailed control over exactly which fields appear.

In some niche cases, you could consider other specialized libraries or even a custom unmarshaller. But generally, these three approaches cover most real-world scenarios.

The Example: Mapping Fields with a Skip Feature

I’ll demonstrate a simplified version of one of my solutions. We’ll define:

• A User struct (representing our internal data).
• A ServiceField list (representing fields exposed by the external API).
• A MatchingTemplate (representing the mapping logic: how each service field is built from an internal field, a fixed value, or marked to skip).

Then we’ll produce a map[string]interface{} representing the final request body.

Let’s walk through the code step by step.

1. Defining the Core Structures

package main

import (
    "encoding/json"
    "errors"
    "fmt"
)
// Represents the user data we have internally
type User struct {
    Email   string `json:"email"`
    Name    string `json:"name"`
    Surname string `json:"surname"`
}
// Represents a field available in the external service
type ServiceField struct {
    Name string `json:"name"`
}
// Returns all service fields that the external API can handle
func getServiceFields() []ServiceField {
    return []ServiceField{
        {Name: "name"},
        {Name: "surname"},
        {Name: "email"},
        {Name: "timezone"},
    }
}

Here, User is our internal representation. In real-world cases, it might have many fields. For demonstration, I’ve limited it to Email, Name, and Surname.

The ServiceField struct identifies fields that the external API expects or supports. In this sample, the API recognizes four: name, surname, email, and timezone.

2. Defining the Template

We want the flexibility to say how each external field is sourced. Some might come from the user ("user"), some might be a fixed string ("fixed"), and others we might want to skip (which we’ll handle later).

type Field struct {
    Source string `json:"source"`
    Value  string `json:"value"`
}

type MatchingTemplate map[string]Field

func getTemplate() MatchingTemplate {
    return MatchingTemplate{
        "email": Field{
            Source: "user",
            Value:  "user.email",
        },
        "name": Field{
            Source: "user",
            Value:  "user.name",
        },
        "surname": Field{
            Source: "user",
            Value:  "user.surname",
        },
        "timezone": Field{
            Source: "fixed",
            Value:  "Asia/Tokyo",
        },
    }
}

• Field.Source indicates where the data is coming from: user, fixed, or skip.
• Field.Value indicates the path or literal value. For "user", the Value might be "user.email" or "user.name". For "fixed", it could be "Asia/Tokyo" or something else.
• If we had a skip scenario, we’d do Field{ Source: "skip" }, which means we don’t include that field at all.

MatchingTemplate is just a map: the keys are the external field names, the values are how to populate them.

3. Processing an Attribute

Next, we need a function that, given an external field name (e.g., "email"), looks up the template and returns either:

1. The value for that field, or
2. An indication that we should skip it, or
3. An error if the field doesn’t exist.

func ProcessAttribute(matchingTemplate MatchingTemplate, serviceAttr string, user User) (string, bool, error) {
    var val string
    field, exists := matchingTemplate[serviceAttr]
    if !exists {
        return "", false, errors.New(serviceAttr + " field not found")
    }
    if field.Source == "skip" {
        // If marked skip, return with skip=true
        return "", true, nil
    } else if field.Source == "fixed" {
        // If fixed, just use the literal value
        val = field.Value
    } else if field.Source == "user" {
        // If from user, map the "Value" string to a field in the user struct
        switch field.Value {
        case "user.name":
            val = user.Name
        case "user.surname":
            val = user.Surname
        case "user.email":
            val = user.Email
        }
    }
    return val, false, nil
}

A few things to highlight:

• We return (string, bool, error) so we can signal an error, or signal that we should skip a field.
• Checking field.Source is how we branch our logic.
• For real code, you might want to handle more elaborate mappings, do validation, or support additional sources.

4. Building the Request Map

Now, let’s piece it all together. In CreateBaseUserRequest, we:

1. Get the template from getTemplate().
2. Get the list of possible service fields.
3. Process each field, skipping where needed.
4. Build a map of the final request object.

func CreateBaseUserRequest(user User) (map[string]interface{}, []error) {
    var processingErrors []error
    template := getTemplate()
    requestMap := map[string]interface{}{}
    serviceFields := getServiceFields()
    for _, sf := range serviceFields {
        res, skip, err := ProcessAttribute(template, sf.Name, user)
        if err != nil {
            processingErrors = append(processingErrors, err)
        }
        if !skip {
            requestMap[sf.Name] = res
        }
    }
    return requestMap, processingErrors
}

• requestMap is where we store the final key-value pairs that we plan to send to the external service.
• If skip == true, we simply don’t add that field.
• Any encountered errors are collected in processingErrors.

5. Putting It All Together in main()

Finally, let’s see it in action:

func main() {
    user := User{
        Name:    "John",
        Surname: "Doe",
        Email:   "test@mail.com",
    }
    request, errs := CreateBaseUserRequest(user)
    if len(errs) > 0 {
        fmt.Println("Errors:", errs)
    }
    fmt.Printf("%v\n", request)  // Print the map
    requestBodyJSON, _ := json.Marshal(request)
    fmt.Printf("%v\n", string(requestBodyJSON))
}

Running this code would produce something like:

map[email:test@mail.com name:John surname:Doe timezone:Asia/Tokyo]
{"email":"test@mail.com","name":"John","surname":"Doe","timezone":"Asia/Tokyo"}

If we introduced a skip example (say, in the template we defined "timezone": {Source: "skip", Value: ""}), then "timezone" would simply be left out of the resulting JSON.

Alternative Approaches

As mentioned earlier, there are a few other ways to handle this:

1. Marshal Struct → Unmarshal to Map → Remove Fields

• If you already have a struct with all possible fields, you could JSON-marshal that struct, unmarshal into a map[string]interface{}, and then prune any keys you don’t want.
• This can be neat if you rely on struct tags for validations or defaults. But it does mean more conversions and overhead.

1. Reflection

• For advanced scenarios, you can use the reflect package to iterate over your struct’s fields by tag or by name. You’d compare each field against a skip-list or user preference, building a map on the fly.
• Reflection can be slower and more verbose. But if you need to keep your definitions in struct form for other reasons (e.g., code generation, schema checks), reflection is a viable approach.

1. Custom JSON Marshal

• You can embed your logic in MarshalJSON() methods on your struct, conditionally removing/adding fields. But this can get complicated if the logic depends on external user choices.

For me, constructing a map is the simplest, most direct approach. It’s flexible and it keeps the code easy to follow.

Conclusion

When you need to dynamically skip fields and map them from various sources in Go, you essentially have two big tasks: where does the data come from? and should we include it or not? By keeping a template that states how each external field is built — and a function to process each one — you get a transparent, easily extensible design.

This pattern works whether you’re building a direct integration with something like the SCIM spec, or a custom RESTful API for provisioning. Of course, your real-world code might be more elaborate. You might have more advanced fields, handle sub-objects, or even interpret special formatting rules. But the core principle remains the same.

I hope this helps anyone wrestling with dynamic request composition in Go. If you have any questions or your own tips to share, feel free to leave a comment below!