GP-grid towards version 1.0.0

Everything started with a simple request from one of my colleagues (thanks, Nancy!):

"Why don’t you create a Grafana plugin that uses gp-grid to render tables?"

I immediately saw it as a fantastic use case for gp-grid, one that could help many people struggling with huge tables, both in terms of rows and columns, in Grafana.

Thanks to gp-grid’s small bundle size, the resulting plugin could remain lightweight and easy to integrate.

But then I faced the harsh truth.

I had been designing gp-grid with simplicity of use in mind. Most of the time, that approach works extremely well. Sometimes, though, like in this case, it comes back to bite you.

  // @gp-grid/core — the data contract today
  export interface DataSourceResponse<TData = unknown> {
    rows: TData[];      // always a list of row objects
    totalRows: number;
  }

  export interface DataSource<TData = unknown> {
    query(request: DataSourceRequest): Promise<DataSourceResponse<TData>>;
    destroy?: () => void;
  }

  export function createClientDataSource<TData = unknown>(
    data: TData[],
    options?: ClientDataSourceOptions<TData>,
  ): DataSource<TData>;

  // @gp-grid/react — pass rows directly, or through a data source
  <Grid columns={columns} rowData={rows} />
  <Grid columns={columns} dataSource={createClientDataSource(rows)} />

As of today, gp-grid accepts data, either through a plain prop or a client-side data source, as a list of rows.

Grafana, on the other hand, manages datasets in a columnar format.

Supporting Grafana with the current gp-grid API would therefore mean taking the entire dataset, an O(n) operation, and converting it into a row-oriented representation before the grid could consume it.

flowchart TD
    App["Application columnar data<br/>arrays / typed arrays"] -->|"O(N × c) copy:<br/>build one object per row"| Objs["TData[] row objects"]
    Objs --> DS["DataSource.query()"]
    DS -->|"{ rows: TData[], totalRows }"| RDM["RowDataManager<br/>row cache of records"]
    RDM --> Cell["getCellValue(row, col)<br/>reads record[field]"]
    RDM --> SF["Sort / filter<br/>read the cached records"]
    RDM --> Slot["SlotData.rowData: TData<br/>always a record"]
    Slot --> R["Renderers<br/>params.rowData[field]"]
    Cell --> Edit["Edit / fill / paste<br/>write into the record"]

  import type { DataFrame } from "@grafana/data";

  // Grafana hands us columns: frame.fields[i].values holds every value of one column.
  // To feed gp-grid today, we must rebuild the whole table as row objects.
  function frameToRows(frame: DataFrame): Record<string, unknown>[] {
    const rows = new Array(frame.length);
    for (let r = 0; r < frame.length; r++) {
      const row: Record<string, unknown> = {};
      for (const field of frame.fields) {
        row[field.name] = field.values[r];
      }
      rows[r] = row;
    }
    return rows; // one new object per row, N × c writes, before a single cell is drawn
  }

  <Grid columns={columns} dataSource={createClientDataSource(frameToRows(frame))} />

That is far from ideal, especially when you consider that gp-grid inside Grafana is meant to handle very large datasets.

The more I thought about it, the more the architectural problem became clear:

Instead of forcing data into the format gp-grid expects, gp-grid should be able to consume the data in the format applications already have.

That changes the problem completely.

Instead of introducing an extra transformation layer between Grafana and gp-grid, the grid should be able to read directly from the underlying columnar representation.

So I thought: why work around the problem when I can use it as an opportunity to improve gp-grid itself?

I decided to take this chance to bring gp-grid to version 1.x, starting with support for zero-copy columnar data sources (read-only, of course).

flowchart TD
    App["Application columns<br/>arrays / typed-array views / getValue(sourceRow)"] -->|"borrowed by reference<br/>O(c) validation, no copy"| CDS

    subgraph CDS["createColumnarDataSource (kind: columnar, writable: false)"]
        Acc["ColumnarAccess<br/>getValue(sourceRow, field)<br/>getRowId: yours or the source position"]
        Q{"query()<br/>sort or filter?"}
        Q -->|"no"| Id["identity RowAccess<br/>view row = source row"]
        Q -->|"yes"| Proj["view RowAccess over an<br/>O(n) index projection<br/>release() frees it"]
        Id & Proj --> Acc
    end

    ObjDS["Object DataSource<br/>(unchanged)"] -->|"{ rows: TData[] }"| RS
    CDS -->|"{ rows: [], totalRows, access }"| RS

    subgraph RDM["RowDataManager"]
        RS["RowStore<br/>access bound? read it : read the row cache"]
    end

    RS --> Cell["getCellValue / getFieldValue<br/>one scalar per visible cell"]
    RS --> SF["SortFilterManager<br/>distinct values read through the access"]
    RS --> Slot["SlotData.rowData: TData | undefined<br/>undefined for columnar rows"]
    Slot --> R["Renderers<br/>value, rowId, getValue(field)"]
    Cell -.->|"not writable"| Rej["Edit / fill / paste rejected<br/>onWriteRejected"]
    CDS -.->|"explicit, O(c) per call"| Rec["getRecord(sourceRow)"]
    CDS -.->|"in-place update"| Rev["setRevision(rev, rowCount?)<br/>then refresh"]

With this approach, gp-grid will no longer require the entire dataset to be converted into rows before rendering. The data can remain in its original columnar representation, while gp-grid accesses only what it needs.

This is particularly important for large datasets, where avoiding unnecessary allocations and transformations can make a significant difference in both memory usage and processing overhead.

And columnar support is only one part of the 1.x release. It will also introduce several other features that have been on the roadmap:

1. Column pinning on the left or right, including RTL (right-to-left) support
2. Column virtualization
3. Row height support
4. Column grouping
5. Pivoting with a client-side aggregation engine
6. And much more!

Once the 1.x release is ready, the next step will be the Grafana plugin itself.

The goal is to enable people to efficiently visualize, filter, sort, and explore huge datasets in Grafana without forcing unnecessary data transformations along the way.

To me, this is a great example of how gp-grid is driven by its community and by passionate people around it.

Sometimes, a small question, suggestion, or doubt is enough to uncover an important use case, challenge an architectural assumption, and eventually turn into a feature that benefits everyone.

That is exactly the kind of feedback that helps gp-grid grow.

If you are curious and want to follow the progress toward the 1.x release, you can track the milestone on GitHub.

Stay tuned, there's a lot more to come!