Server Row Data Source

Many datasets are massive and are too large to load all at once into a user's browser. In order to maintain performance, we need to ensure we load only what is needed - keeping most of the data on the server. For this purpose, the server row data source is the perfect fit.

LyteNyte Grid's server row data source allows data to be partially loaded - ensuring only what is necessary is sent to the browser.

As its name suggests the server data source will request data from an external source - normally your own server. The responses expected must conform to an interface defined by the LyteNyte Grid server data source.

Creating A Server Row Data Source

LyteNyte Grid exports a useServerDataSource hook to create a server data source. At a minimum a dataFetcher function must be provided. The dataFetcher function is responsible for requesting data from a server. The example below shows the most basic server data source setup.

TODO;

The Data Fetcher

The dataFetcher function provided to the server data source is the work horse of the data source. The function will receive an array of AsyncDataRequestBlock objects. An AsyncDataRequestBlock is a description of the block of data the server should send back to the data source. It has the following interface:

export interface AsyncDataRequestBlock {
  readonly id: string;
  readonly blockKey: number;
  readonly path: string[];
  readonly rowStart: number;
  readonly rowEnd: number;
  readonly blockStart: number;
  readonly blockEnd: number;
}

Each field is explained in turn:

• id is a unique identifier for this block. Primarily used by LyteNyte Grid, but you can leverage it to uniquely identify blocks
• blockKey is a number representing the block offset. The server data source partitions the row count into blocks, for example if there are 1000 rows, it may partition these rows into blocks of 1000, making a total of 10 blocks. The blockKey is used to mark a partition index. A blockKey of 1 would correspond to the rows starting from 100 (block keys count from 0)
• path: a path for this block. This field is only populated when their are row groups present. The path is a string of parent keys. An empty path indicates the block is at the root level
• rowStart is the first row index for this block
• rowEnd is the last row index for this block
• blockStart is the first block index. For example if the blockKey is 2, then the blockStart would be 200 - if the block sizes are 100
• blockEnd is the last block index for this block

The server data source may request an array of blocks depending on the current view configuration. Row grouping and pivots tend to create more blocks whereas flatter views usually only request one or two blocks.

Data Fetcher Response

Given the AsyncDataRequestBlocks requested that dataFetcher function should return an AsyncDataResponse object.

export type AsyncDataResponse = {
  readonly rootCount?: number;
  readonly reqTime: number;
  readonly blocks: AsyncDataBlock[];
  readonly topBlock?: AsyncDataBlockPinned;
  readonly bottomBlock?: AsyncDataBlockPinned;
  readonly totalBlock?: AsyncDataBlockTotal;
};

The AsyncDataResponse object may return more than what was requested but at a minimum it should return a response for each requested block. The response should be present in the blocks property.

Here we cover each property of the AsyncDataResponse object. The rootCount property is used set the number of root rows in the grid. When there is no grouping involved the rootCount should be the same as the total rowCount. When row groups are set, a hierarchy is formed with some rows being considered children of other rows. A row that does not have a parent is considered a root row. In this case the rootCount should be the number of rows that do not have parents, i.e. the count of the top most rows when row groups are present.

The reqTime is a unix timestamp when the request was sent. Your server implementation should return the this. The dataFetcher args receive the reqTime as a parameter. It is used internally by the server data source to ignore outdated requests.

The blocks property is an array of responses for the requested blocks. It may also contain more blocks than requests - for optimistically loading data. The response should be an array AsyncDataBlock object, which has the following interface:

export type AsyncDataBlock = {
  readonly blockKey: number;
  readonly frame: {
    readonly data: unknown[];
    readonly ids: string[];
    readonly pathKeys: (string | null)[];
    readonly kinds: (RowGroupKind | RowLeafKind)[];
    readonly childCounts: number[];
  };
  readonly size: number;
  readonly path?: string[];
};

The fields here can be complex but we cover each one in turn:

• The blockKey is the same as the request blockKey but in response

• The frame object is the data for the async block. Notice how each field is an array. The server data source will use this to create the rows from the response. Each array should have equal length.

  • data are the data associated with the row
  • ids are the id values for each row
  • pathKeys are the paths for the rows
  • kinds is the kind of row the grid should create. These are numbers with RowGroupKind being 2, and RowLeafKind being 1
  • childCounts is the number of child rows for each row. If the row is a leaf row its corresponding child count should be 0

• The size property is the size of the overall block

• The path is the path for this block. If it is a root block then path should be empty or omitted

The remaining fields pertain to the pinned rows and the totals row. The server may optionally response with the following interfaces respectively:

export type AsyncDataBlockPinned = {
  readonly frame: {
    readonly data: unknown[];
    readonly ids: string[];
  };
};
 
export type AsyncDataBlockTotal = {
  readonly frame: {
    readonly data: unknown;
  };
};

In the case of the AsyncDataBlockPinned, like the blocks response, the frame property should have arrays of equal length.

This is a lot of theory and many of the concepts may not seem straightforward. An example will illuminate most of the complexity, and you will find that the server data source interface is actually very simple, and that most of the complexity lies in the data processing layer on the server.

TODO;

Row Grouping Example

Row Grouping increases the number of blocks the server data source has to manage, and in particular increases the variety of responses your server needs to be able to send. The example below shows a basic implementation of row grouping in action.

TODO;

TODO: Row Data Clearing

TODO: rowPathSeparator

TODO: blockSize

TODO: columnPivoting

TODO: columnIn Filtering

TODO: Server Data Limitations