kennylong

Build Elegant APIs(Part 1): Trafic Saving

#Abstract

To reduce network traffic, it is sometimes acceptable for the client to limit the parts of the resource that the server should return in its response, i.e., to return only some fields of the resource rather than the complete resource representation. This article discusses a simple implementation method that does not require defining additional message structures, ensuring flexibility and maintainability.

#Background

For example, in the Douban leaderboard's list and detail pages shown below, the former only needs to display some brief description information, such as title, director, actors, time, and rating, etc. When the user is interested and clicks to enter the latter, it will display more detailed and complete content, usually with much more information.

Alt text Alt text

A common practice is to define a simple resource message, such as ItemBrief for the list page interface, and then return the complete ItemDetail for the detail page, i.e., two message structures. Using protobuf, it might be represented as follows:

Usually, there will be nested complex message structures, but for simplicity, we will not discuss them here.

message ItemBrief {
    string title = 1;
    string director = 2;
    repeated string actors = 3;
    string poster = 4;
}

message ItemDetail {
    string title = 1;
    string director = 2;
    repeated string actors = 3;
    string poster = 4;

    string desc = 5;
    repeated string actors_info = 6;
    string discussion = 7;
    // ....
}

Then, they are used as return values in the ListItem and GetItem RPCs. This approach is more direct but increases cognitive overhead and reduces flexibility.

On the one hand, API users need to know that there are multiple data structures representing the same resource; on the other hand, if the client needs detail information in some scenarios, such as the list interface, but only the get interface has it, it loses flexibility.

Furthermore, in line with the principle that interfaces should be as orthogonal as possible, i.e., interfaces should be as unrelated as possible, without this constraint, similar interfaces are likely to become more and more numerous as requirements evolve, causing confusion for clients and maintenance costs for themselves.

For the above addition and subtraction issues, we propose a better solution that meets both requirements.

#Solution

The answer is to return partial responses, i.e., allowing the client to specify parts of the response resource. Specifically, we only have one resource message:

message Item {
    string id = 1;
    string title = 2;
    string author = 3;
    int32 view_count = 5;
    string create_time = 4;
    string desc = 5;
    string content = 6;
    // ...
}

Then define an enumeration type:

enum ItemView {
    // Not specified, default to BRIEF.
    ITEM_VIEW_UNSPECIFIED = 0;
    // Show only brief fields:
    // - id
    // - title
    // - author
    // - view_count
    // - create_time
    BRIEF = 1;
    // Show all the fields.
    DETAIL = 2;
}

This enumeration represents the resource view, i.e., how the resource is presented:

  • unspecified is the default value, i.e., the user does not specify, and the server acts according to its own way, such as returning a concise resource for the list interface and a complete resource for the get interface.
  • brief is the concise version, returning only some preset fields.
  • detail is the complete version, returning all fields.

Then, in the RPC, add this enumeration field to the request message:

// Get contract comparison task list
service Service {
    rpc ListItem(ListItemRequest) returns (ListItemResponse) {
        option (google.api.http) = {
            get: "/v1/items"
        };
    };

    rpc GetItem(GetItemRequest) returns (Item) {
        option (google.api.http) = {
            get: "/v1/items/{id}"
        };
    };
}

message ListItemRequest {
    int32 page_size = 1;

    // ... other fields

    // view layout
    View view = 2;

    // TODO: field mask!
}

message ListItemResponse {
    repeated Item items = 1;
    int32 total_size = 2;
}

Considering that the Get interface generally returns the complete version, we will only discuss the List interface. The new view parameter allows users to specify whether the resource view returned is brief or complete, and the resource data structure returned by the Get and List interfaces is the same. Moreover, we can even allow the client to choose the fields to return. This achieves simplicity, consistency, and flexibility. Clients do not need to define additional data types for mapping.

Now let's see how to implement this.

#Implementation

In early 2020, Google released a new protocol buffer Golang library that provides reflection APIs, allowing us to recursively traverse the message structure and then check if the field falls within the specified view node. If not, we can use the API to clear its value (set it to zero), thus obtaining the desired field collection.

The specific implementation is as follows (other data structures can be used to store the desired field collection, this is just a simple comparison):

func mask(msg proto.Message) {
    pr := msg.ProtoReflect()
    pr.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
        if fd.Name() != "required fields" { // Check if the field is needed
            if fd.Kind() == protoreflect.MessageKind {
                mask(v.Message().Interface()) // recursive
            }
            return true
        }
        pr.Clear(fd) // Clear the field
        return true
    })
}

Then, call the above function before returning in the RPC.

#Effect

Alt text Alt text

As shown in the figure, in our test environment, with a single-page request list quantity of 1, the content-length of the BRIEF and DETAIL views are 642 and 11069, respectively, reducing the content transmission volume by 94%. The difference will be more pronounced for larger list lengths.

Of course, different business data comparisons will vary greatly, and redefining a simplified message type can also achieve this. The main purpose of this article is to discuss a flexible and maintainable API design that ensures simple and consistent interfaces.

#Compatibility

Since the resource view enumeration View is a new field, it can be compatible with the existing logic by default.

#Going Further

This article discusses server-side preset resource view collections. What if the client-side is not satisfied? Is there a way, like graphql, to allow the client-side to choose the fields they want?

The answer is yes. At this point, you'll need to use field masks, which can also be used to store resource view field collections in this article.

Interested readers can follow the next article, hehe.

#References