Part 1: Querying dApp GraphQL databases in Dart hosted on The Graph

wafrat -

I wanted to read the GraphQL database in Dart to make analytics for the decentralized platform Aave. It turned out to be harder than I thought!

The Graph

The Graph is a service that hosts GraphQL databases for dApps. One specificity is that The Graph adds some annotations to GraphQL schema to automate certain things. More on that later in the paragraph about Ferry.

Querying the data manually is straightforward using their online tool.

Or we can try something even simpler to start.

web3dart

It works but the information of each currency is not public.

GraphQL

The two most popular packages are graphql and ferry.

Ferry

What's attractive about ferry is that it's fully typed. The documentation really sells it well. First you download the schema with some command-line tool, then you generate the data classes, and finally you request the data, and it'll come in all typed for you.

Downloading the schema

I tried downloading the schema using this command:

get-graphql-schema https://api.thegraph.com/subgraphs/name/aave/protocol-v2 > lib/schema.graphql

It ran fine and generated a schema of over 8000 lines.

Adding dependencies

The doc says to install the latest versions of these dependencies and dev dependencies:

But that forces the reader to look up each package on pub.dev. It's quite tedious. Instead, here's how to install it without looking them up individually. To add a dev dependency, use --dev as explained at https://dart.dev/tools/pub/cmd/pub-add:

dart pub add ferry
dart pub add gql_http_link
dart pub add --dev ferry_generator
dart pub add --dev build_runner

I should send a PR to update their doc, if I have time.

Generating classes

The doc at https://pub.dev/packages/ferry#build-generated-classes instructs us to set up a build.yaml file and then run:

pub run build_runner build

However, I get the following error:

zsh: command not found: pub

The command should be:

dart run build_runner build

Yet another PR I could send to improve their doc.

Even after it runs, the tool fails to complete.

And the resulting file has over 1000 errors.

So the conclusion is that get-graphql-schema does not play well with The Graph schema.

Finding an alternate schema

Luckily, the Aave GraphQL schema is publicly available on Aave's own repository: https://github.com/aave/protocol-v2-subgraph/blob/main/schema.graphql. And the file is only 700 lines long.

Now let's generate the classes for it:

dart run build_runner build

This time it generates classes and it compiles fine.

Executing a request

According to the doc, if I write a request definition, I should then be able to instantiate a G[type_name]Req and make a request. I want to fetch Aave's Borrow information.

So let's write a borrows.graphql similar to their example:

query Borrows {
  borrows {
    amount
  }
}

Upon re-generating the classes, I get this error:

[SEVERE] built_value_generator:built_value on lib/borrows.req.gql.dart:
Error in BuiltValueGenerator for abstract class GBorrowsReq implements Built<GBorrowsReq, dynamic>, OperationRequest<dynamic, GBorrowsVars>.
Please make the following changes to use BuiltValue:

1. Make field optimisticResponse have non-dynamic type. If you are already specifying a type, please make sure the type is correctly imported.

It seems that ferry can't figure out the type of the borrows field. In fact, if you look more closely to the schema, you'll find that there is no Query type. Yet in GraphQL's doc, it states:

Every GraphQL service has a query type and may or may not have a mutation type. These types are the same as a regular object type, but they are special because they define the entry point of every GraphQL query.

It turns out that in The Graph, they introduce a notion of entities. An entity in The Graph is a type for which The Graph will automatially add the definition in the generated Query. See https://thegraph.com/docs/en/developer/graphql-api/ and https://thegraph.com/docs/en/developer/create-subgraph-hosted/#defining-entities:

With The Graph, you simply define entity types in schema.graphql, and Graph Node will generate top level fields for querying single instances and collections of that entity type. Each type that should be an entity is required to be annotated with an @entity directive.

So how do I generate that schema myself? Or perhaps I can manually add the Borrow definition to the schema.

Hardcoding the Borrow type

Let's write a quick and dirty schema for borrows at the bottom of schema.graphql.

schema {
  query: Query
}

type Query {
  borrows: [Borrow!]!
}

The tool completes the generation but lib/schema.var.gql.g.dart fails to compile with over a thousand errors. I don't want to mess more with the schema. It feels like it would not be productive. Let's try to get the generated schema. If this method fails too, I'll resort to trying ferry on one of GraphQL's example vanilla databases.

Finding the generated schema

Using The Graph's tutorial

Now I'm looking at https://thegraph.com/docs/en/developer/quick-start/. I should install their command-line tool graph-cli, then run graph codegen and see what happens. This is what happens:

% graph codegen                               
The current version of graph-cli can't be used with mappings on apiVersion less than '0.0.5'

Indeed, in the project's subgraph.yaml, it says:

specVersion: 0.0.2

Let's install an older version of graph-cli. How am I supposed to know which version would work? Let's look at the project's package.json:

"@graphprotocol/graph-cli": "^0.18.0"

Alright, let's run sudo npm install -g @graphprotocol/graph-cli@0.18.0 and try again. This time it ran successfully:

% graph codegen                                      
  Skip migration: Bump mapping apiVersion from 0.0.1 to 0.0.2
  Skip migration: Bump mapping apiVersion from 0.0.2 to 0.0.3
  Skip migration: Bump mapping apiVersion from 0.0.3 to 0.0.4
  Skip migration: Bump mapping specVersion from 0.0.1 to 0.0.2
✔ Apply migrations
✔ Load subgraph from subgraph.yaml
  Load contract ABI from externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json
  Load contract ABI from externals/protocol-v2/artifacts/contracts/mocks/oracle/GenericOracleI.sol/GenericOracleI.json
  Load contract ABI from externals/protocol-v2/artifacts/contracts/mocks/oracle/IExtendedPriceAggregator.sol/IExtendedPriceAggregator.json
  Load contract ABI from externals/protocol-v2/artifacts/contracts/dependencies/openzeppelin/contracts/IERC20Detailed.sol/IERC20Detailed.json
  Load contract ABI from constant-abis/EACAggregatorProxy.json
  Load contract ABI from constant-abis/OracleAnchor.json
  Load contract ABI from externals/protocol-v2/artifacts/contracts/dependencies/openzeppelin/contracts/IERC20Detailed.sol/IERC20Detailed.json
  Load contract ABI from externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json
  Load contract ABI from externals/protocol-v2/artifacts/contracts/mocks/oracle/IExtendedPriceAggregator.sol/IExtendedPriceAggregator.json
  Load contract ABI from constant-abis/EACAggregatorProxy.json
  Load contract ABI from constant-abis/ChainlinkSourcesRegistry.json
  Load contract ABI from externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json
  Load contract ABI from externals/protocol-v2/artifacts/contracts/mocks/oracle/GenericOracleI.sol/GenericOracleI.json
  Load contract ABI from externals/protocol-v2/artifacts/contracts/mocks/oracle/IExtendedPriceAggregator.sol/IExtendedPriceAggregator.json
  Load contract ABI from constant-abis/ENSResolverChainlink.json
  Load contract ABI from externals/protocol-v2/artifacts/contracts/mocks/oracle/IExtendedPriceAggregator.sol/IExtendedPriceAggregator.json
  Load contract ABI from externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json
  Load contract ABI from externals/protocol-v2/artifacts/contracts/protocol/configuration/LendingPoolAddressesProviderRegistry.sol/LendingPoolAddressesProviderRegistry.json
  Load contract ABI from externals/protocol-v2/artifacts/contracts/adapters/UniswapLiquiditySwapAdapter.sol/UniswapLiquiditySwapAdapter.json
  Load contract ABI from externals/protocol-v2/artifacts/contracts/adapters/UniswapRepayAdapter.sol/UniswapRepayAdapter.json
✔ Load contract ABIs
  Generate types for contract ABI: AaveOracle (externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json)
  Write types to generated/AaveOracle/AaveOracle.ts
  Generate types for contract ABI: GenericOracleI (externals/protocol-v2/artifacts/contracts/mocks/oracle/GenericOracleI.sol/GenericOracleI.json)
  Write types to generated/AaveOracle/GenericOracleI.ts
  Generate types for contract ABI: IExtendedPriceAggregator (externals/protocol-v2/artifacts/contracts/mocks/oracle/IExtendedPriceAggregator.sol/IExtendedPriceAggregator.json)
  Write types to generated/AaveOracle/IExtendedPriceAggregator.ts
  Generate types for contract ABI: IERC20Detailed (externals/protocol-v2/artifacts/contracts/dependencies/openzeppelin/contracts/IERC20Detailed.sol/IERC20Detailed.json)
  Write types to generated/AaveOracle/IERC20Detailed.ts
  Generate types for contract ABI: EACAggregatorProxy (constant-abis/EACAggregatorProxy.json)
  Write types to generated/AaveOracle/EACAggregatorProxy.ts
  Generate types for contract ABI: OracleAnchor (constant-abis/OracleAnchor.json)
  Write types to generated/OracleAnchor/OracleAnchor.ts
  Generate types for contract ABI: IERC20Detailed (externals/protocol-v2/artifacts/contracts/dependencies/openzeppelin/contracts/IERC20Detailed.sol/IERC20Detailed.json)
  Write types to generated/OracleAnchor/IERC20Detailed.ts
  Generate types for contract ABI: AaveOracle (externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json)
  Write types to generated/OracleAnchor/AaveOracle.ts
  Generate types for contract ABI: IExtendedPriceAggregator (externals/protocol-v2/artifacts/contracts/mocks/oracle/IExtendedPriceAggregator.sol/IExtendedPriceAggregator.json)
  Write types to generated/OracleAnchor/IExtendedPriceAggregator.ts
  Generate types for contract ABI: EACAggregatorProxy (constant-abis/EACAggregatorProxy.json)
  Write types to generated/OracleAnchor/EACAggregatorProxy.ts
  Generate types for contract ABI: ChainlinkSourcesRegistry (constant-abis/ChainlinkSourcesRegistry.json)
  Write types to generated/ChainlinkSourcesRegistry/ChainlinkSourcesRegistry.ts
  Generate types for contract ABI: AaveOracle (externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json)
  Write types to generated/ChainlinkSourcesRegistry/AaveOracle.ts
  Generate types for contract ABI: GenericOracleI (externals/protocol-v2/artifacts/contracts/mocks/oracle/GenericOracleI.sol/GenericOracleI.json)
  Write types to generated/ChainlinkSourcesRegistry/GenericOracleI.ts
  Generate types for contract ABI: IExtendedPriceAggregator (externals/protocol-v2/artifacts/contracts/mocks/oracle/IExtendedPriceAggregator.sol/IExtendedPriceAggregator.json)
  Write types to generated/ChainlinkSourcesRegistry/IExtendedPriceAggregator.ts
  Generate types for contract ABI: ChainlinkENSResolver (constant-abis/ENSResolverChainlink.json)
  Write types to generated/ChainlinkENSResolver/ChainlinkENSResolver.ts
  Generate types for contract ABI: IExtendedPriceAggregator (externals/protocol-v2/artifacts/contracts/mocks/oracle/IExtendedPriceAggregator.sol/IExtendedPriceAggregator.json)
  Write types to generated/ChainlinkENSResolver/IExtendedPriceAggregator.ts
  Generate types for contract ABI: AaveOracle (externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json)
  Write types to generated/ChainlinkENSResolver/AaveOracle.ts
  Generate types for contract ABI: LendingPoolAddressesProviderRegistry (externals/protocol-v2/artifacts/contracts/protocol/configuration/LendingPoolAddressesProviderRegistry.sol/LendingPoolAddressesProviderRegistry.json)
  Write types to generated/LendingPoolAddressesProviderRegistry/LendingPoolAddressesProviderRegistry.ts
  Generate types for contract ABI: UniswapLiquiditySwapAdapter (externals/protocol-v2/artifacts/contracts/adapters/UniswapLiquiditySwapAdapter.sol/UniswapLiquiditySwapAdapter.json)
  Write types to generated/UniswapLiquiditySwapAdapter/UniswapLiquiditySwapAdapter.ts
  Generate types for contract ABI: UniswapRepayAdapter (externals/protocol-v2/artifacts/contracts/adapters/UniswapRepayAdapter.sol/UniswapRepayAdapter.json)
  Write types to generated/UniswapRepayAdapter/UniswapRepayAdapter.ts
✔ Generate types for contract ABIs
  Generate types for data source template AaveIncentivesController
  Generate types for data source template FallbackPriceOracle
  Generate types for data source template ChainlinkAggregator
  Generate types for data source template AToken
  Generate types for data source template StableDebtToken
  Generate types for data source template VariableDebtToken
  Generate types for data source template LendingPoolAddressesProvider
  Generate types for data source template LendingPoolConfigurator
  Generate types for data source template LendingPool
  Generate types for data source template UniswapExchange
  Generate types for data source template BalancerPool
  Write types for templates to generated/templates.ts
✔ Generate types for data source templates
  Load data source template ABI from constant-abis/AaveIncentivesController.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/dependencies/openzeppelin/contracts/IERC20Detailed.sol/IERC20Detailed.json
  Load data source template ABI from constant-abis/AToken.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/mocks/oracle/GenericOracleI.sol/GenericOracleI.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/mocks/oracle/IExtendedPriceAggregator.sol/IExtendedPriceAggregator.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json
  Load data source template ABI from constant-abis/AToken.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/dependencies/openzeppelin/contracts/IERC20Detailed.sol/IERC20Detailed.json
  Load data source template ABI from constant-abis/AaveIncentivesController.json
  Load data source template ABI from constant-abis/StableDebtToken.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/dependencies/openzeppelin/contracts/IERC20Detailed.sol/IERC20Detailed.json
  Load data source template ABI from constant-abis/AaveIncentivesController.json
  Load data source template ABI from constant-abis/VariableDebtToken.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/dependencies/openzeppelin/contracts/IERC20Detailed.sol/IERC20Detailed.json
  Load data source template ABI from constant-abis/AaveIncentivesController.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/protocol/configuration/LendingPoolAddressesProvider.sol/LendingPoolAddressesProvider.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/protocol/lendingpool/LendingPoolConfigurator.sol/LendingPoolConfigurator.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/dependencies/openzeppelin/contracts/IERC20Detailed.sol/IERC20Detailed.json
  Load data source template ABI from constant-abis/IERC20DetailedBytes.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/protocol/lendingpool/DefaultReserveInterestRateStrategy.sol/DefaultReserveInterestRateStrategy.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/protocol/lendingpool/LendingPool.sol/LendingPool.json
  Load data source template ABI from constant-abis/IUniswapExchange.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json
  Load data source template ABI from constant-abis/IBalancerPool.json
  Load data source template ABI from externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json
✔ Load data source template ABIs
  Generate types for data source template ABI: AaveIncentivesController > AaveIncentivesController (constant-abis/AaveIncentivesController.json)
  Write types to generated/templates/AaveIncentivesController/AaveIncentivesController.ts
  Generate types for data source template ABI: AaveIncentivesController > IERC20Detailed (externals/protocol-v2/artifacts/contracts/dependencies/openzeppelin/contracts/IERC20Detailed.sol/IERC20Detailed.json)
  Write types to generated/templates/AaveIncentivesController/IERC20Detailed.ts
  Generate types for data source template ABI: AaveIncentivesController > AToken (constant-abis/AToken.json)
  Write types to generated/templates/AaveIncentivesController/AToken.ts
  Generate types for data source template ABI: FallbackPriceOracle > GenericOracleI (externals/protocol-v2/artifacts/contracts/mocks/oracle/GenericOracleI.sol/GenericOracleI.json)
  Write types to generated/templates/FallbackPriceOracle/GenericOracleI.ts
  Generate types for data source template ABI: FallbackPriceOracle > AaveOracle (externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json)
  Write types to generated/templates/FallbackPriceOracle/AaveOracle.ts
  Generate types for data source template ABI: ChainlinkAggregator > IExtendedPriceAggregator (externals/protocol-v2/artifacts/contracts/mocks/oracle/IExtendedPriceAggregator.sol/IExtendedPriceAggregator.json)
  Write types to generated/templates/ChainlinkAggregator/IExtendedPriceAggregator.ts
  Generate types for data source template ABI: ChainlinkAggregator > AaveOracle (externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json)
  Write types to generated/templates/ChainlinkAggregator/AaveOracle.ts
  Generate types for data source template ABI: AToken > AToken (constant-abis/AToken.json)
  Write types to generated/templates/AToken/AToken.ts
  Generate types for data source template ABI: AToken > IERC20Detailed (externals/protocol-v2/artifacts/contracts/dependencies/openzeppelin/contracts/IERC20Detailed.sol/IERC20Detailed.json)
  Write types to generated/templates/AToken/IERC20Detailed.ts
  Generate types for data source template ABI: AToken > AaveIncentivesController (constant-abis/AaveIncentivesController.json)
  Write types to generated/templates/AToken/AaveIncentivesController.ts
  Generate types for data source template ABI: StableDebtToken > StableDebtToken (constant-abis/StableDebtToken.json)
  Write types to generated/templates/StableDebtToken/StableDebtToken.ts
  Generate types for data source template ABI: StableDebtToken > IERC20Detailed (externals/protocol-v2/artifacts/contracts/dependencies/openzeppelin/contracts/IERC20Detailed.sol/IERC20Detailed.json)
  Write types to generated/templates/StableDebtToken/IERC20Detailed.ts
  Generate types for data source template ABI: StableDebtToken > AaveIncentivesController (constant-abis/AaveIncentivesController.json)
  Write types to generated/templates/StableDebtToken/AaveIncentivesController.ts
  Generate types for data source template ABI: VariableDebtToken > VariableDebtToken (constant-abis/VariableDebtToken.json)
  Write types to generated/templates/VariableDebtToken/VariableDebtToken.ts
  Generate types for data source template ABI: VariableDebtToken > IERC20Detailed (externals/protocol-v2/artifacts/contracts/dependencies/openzeppelin/contracts/IERC20Detailed.sol/IERC20Detailed.json)
  Write types to generated/templates/VariableDebtToken/IERC20Detailed.ts
  Generate types for data source template ABI: VariableDebtToken > AaveIncentivesController (constant-abis/AaveIncentivesController.json)
  Write types to generated/templates/VariableDebtToken/AaveIncentivesController.ts
  Generate types for data source template ABI: LendingPoolAddressesProvider > LendingPoolAddressesProvider (externals/protocol-v2/artifacts/contracts/protocol/configuration/LendingPoolAddressesProvider.sol/LendingPoolAddressesProvider.json)
  Write types to generated/templates/LendingPoolAddressesProvider/LendingPoolAddressesProvider.ts
  Generate types for data source template ABI: LendingPoolConfigurator > LendingPoolConfigurator (externals/protocol-v2/artifacts/contracts/protocol/lendingpool/LendingPoolConfigurator.sol/LendingPoolConfigurator.json)
  Write types to generated/templates/LendingPoolConfigurator/LendingPoolConfigurator.ts
  Generate types for data source template ABI: LendingPoolConfigurator > IERC20Detailed (externals/protocol-v2/artifacts/contracts/dependencies/openzeppelin/contracts/IERC20Detailed.sol/IERC20Detailed.json)
  Write types to generated/templates/LendingPoolConfigurator/IERC20Detailed.ts
  Generate types for data source template ABI: LendingPoolConfigurator > IERC20DetailedBytes (constant-abis/IERC20DetailedBytes.json)
  Write types to generated/templates/LendingPoolConfigurator/IERC20DetailedBytes.ts
  Generate types for data source template ABI: LendingPoolConfigurator > DefaultReserveInterestRateStrategy (externals/protocol-v2/artifacts/contracts/protocol/lendingpool/DefaultReserveInterestRateStrategy.sol/DefaultReserveInterestRateStrategy.json)
  Write types to generated/templates/LendingPoolConfigurator/DefaultReserveInterestRateStrategy.ts
  Generate types for data source template ABI: LendingPool > LendingPool (externals/protocol-v2/artifacts/contracts/protocol/lendingpool/LendingPool.sol/LendingPool.json)
  Write types to generated/templates/LendingPool/LendingPool.ts
  Generate types for data source template ABI: UniswapExchange > IUniswapExchange (constant-abis/IUniswapExchange.json)
  Write types to generated/templates/UniswapExchange/IUniswapExchange.ts
  Generate types for data source template ABI: UniswapExchange > AaveOracle (externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json)
  Write types to generated/templates/UniswapExchange/AaveOracle.ts
  Generate types for data source template ABI: BalancerPool > IBalancerPool (constant-abis/IBalancerPool.json)
  Write types to generated/templates/BalancerPool/IBalancerPool.ts
  Generate types for data source template ABI: BalancerPool > AaveOracle (externals/protocol-v2/artifacts/contracts/misc/AaveOracle.sol/AaveOracle.json)
  Write types to generated/templates/BalancerPool/AaveOracle.ts
✔ Generate types for data source template ABIs
✔ Load GraphQL schema from schema.graphql
  Write types to generated/schema.ts
✔ Generate types for GraphQL schema

Types generated successfully

Looks like all it generated are TypeScript definitions of entities.

Too bad. And then there are errors in the generated schema.ts. Let's try the second command in the tutorial: graph build...

This fails too.

Another way would be to look at Aave protocol's doc

They actually have prepared a bunch of scripts. All I need to do is run them.

This requires I make an account on The Graph and create a test graph. Much more heavyweight that I had assumed. Also, prepare:all uses a yaml template to generate the actual file that gets fed to graph build.

After running for a while, prepare:all fails with a similar error:

Failed to load subgraph from subgraph.yaml: Error in subgraph.yaml: Event with signature 'WethSet(indexed address)' not present in ABI 'AaveOracle'.

Turns out it's because of a change in an external repository. Using the suggested fix, compilation works correctly.

No more compilation error in `schema.ts`

But again, this does not generate the final GraphQL schema, just the TypeScript definitions.

Looking at the docs on npm, that's indeed what the tool does:

graph codegen — Generates AssemblyScript types for smart contract ABIs and the subgraph schema.
graph build — Compiles a subgraph to WebAssembly.

After exploring the graph-cli repository, I found a tool that does what I want. https://github.com/graphprotocol/graph-node/blob/master/graphql/examples/schema.rs:

println!("\nPrint the API schema we derive from the given input schema");

The script is written in Rust, which I have never used. Also it looks like I should first compile their library. The package definition is done in a file called Cargo.toml, which appears to be the standard Rust package definition file. It really feels like I'm on a wild goose hunt.

Anyway, let's keep going and learn a bit of Rust and Cargo. At least the equivalent of npm i and npm run...

Ok, after going through their Getting started guide, I'm ready for the next challenge:

  1. git clone https://github.com/graphprotocol/graph-node.git
  2. cd graph-node/graphql
  3. cargo build

However, the command builds the reusable package, not the example file.

How do I run examples/schema.rs? rustc examples/schema.rs returns this error. It can't find the create I just built:

Based on https://doc.rust-lang.org/book/ch07-01-packages-and-crates.html, it looks like I can rename this into src/main.rs, and my crate become a binary and a package at the same time.

However, when I tried to compile it again, I got this error:

Compiling graph-graphql v0.25.0 (/Users/anhtuan/git/graph-node/graphql)
error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-arch" "arm64" [...]
  [...]
  = note: ld: library not found for -lpq
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

What's this? Using those three posts:

I installed postgresql with brew install postgresql. The install script then told me to run brew services restart postgresql to start the service. I don't know if I need to. Probably not, but I ran it anyway. Then cargo build was still failing, so I ran cargo clean followed by cargo build. This time it ran fine. And one thing I hadn't noticed before, but the target/ folder in which Cargo compiles everything was generated in the repository's root folder, not in the graphql subfolder. So while I was in the graphql subfolder, I ran:

graphql % ./../target/debug/graph-graphql 
please provide a GraphQL schema
usage: schema schema.graphql

Print the API schema we derive from the given input schema

Finally!!! Btw, I wonder if renaming the example file was really necessary. Perhaps I had just not looked properly into the target folder.

Let's try it on the Aave subgraph schema:

% graph-node/target/debug/graph-graphql protocol-v2-subgraph/schema.graphql > full_schema.graphql

The resulting file is nearly 5000 lines, and finally contains the Query type. It also contains the definition for borrows:

type Query {
  ...
  borrows(skip: Int = 0, first: Int = 100, orderBy: Borrow_orderBy, orderDirection: OrderDirection, where: Borrow_filter, "The block at which the query should be executed. Can either be a `{ hash: Bytes }` value containing a block hash, a `{ number: Int }` containing the block number, or a `{ number_gte: Int }` containing the minimum block number. In the case of `number_gte`, the query will be executed on the latest block only if the subgraph has progressed to or past the minimum block number. Defaults to the latest block when omitted." block: Block_height, "Set to `allow` to receive data even if the subgraph has skipped over errors while syncing." subgraphError: _SubgraphErrorPolicy_! = deny): [Borrow!]!
  ...
}

Hallelujah! Let's put that file in my Dart project and re-generate the classes.

Unfortunately, the code generation for the schema fails, and that's before I even write the request file.

Now I have two hypotheses:

  1. I used the raw protocol-v2-subgraph/schema.graphql instead of the post-processed schema based off subgraph.template.yaml.
  2. The code builder can't parse such a long or complicated schema.

Let's check the first hypothesis. Aave's protocol-v2-subgraph/package.json shows that it doesn't re-generate a schema.graphql. Rather it generates a schema.yaml with hardcoded values, based on the Ethereum environment that the user wants to deploy the subgraph to. schema.yaml matters only when hosting the subgraph. The GraphQL schema definition itself is still schema.graphql. See https://thegraph.com/docs/en/developer/create-subgraph-hosted/.

So my guess is that the code builder just can't process that one schema. Let's look at the error a little closer. One of the errors is

line 69277, column 17645 of .: Expected an identifier.

In .dart_tool/build/generated/console_simple/lib/schema.schema.gql.built_value.g.part, we have:

Checking the type in graphql.schema, there is really nothing out of the ordinary in Reserve_filter...

...besides the fact that it has 630 properties. Since I don't plan to use this filter, at least in the beginning, let's trim down the type and see what the code generation tool does with it.

And lo and behold, the tool actually ran without an error, and the resulting Dart files compile. Let's restore the query file borrows.graphql and try again.

Not sure what the error means. Is it because the name of my query conflicts with the name of an actual type?

Even after renaming my Borrows query to MyBorrows, I get the same error message, so that's not the cause of the issue.

My next stop is ferry's own repository and example project. In their project, they query a list of all available Pokemon. See their all_pokemon.graphql query definition here.

Their generated file examples/pokemon_explorer/lib/src/graphql/__generated__/all_pokemon.data.gql.dart contains the same instantiation as in mine, but the compiler does not complain that string is not the expected type.

What gives? It looks like the types match in their example:

The `G__typename` setter in GAllPokemonDataBuilder expects a String

But my own expects a StringBuilder.

Smells like mismatching versions somewhere. Let's compare the example dependency versions vs mine.

name: pokemon_explorer
...
dependencies:
  gql_link: ^0.4.0-nullsafety.3
  gql_http_link: ^0.4.0-nullsafety.1
  ferry: ^0.10.1-0.1.nullsafety.0
  ferry_hive_store: ^0.4.5-dev.1
  ferry_flutter: ^0.5.5-dev.2
  ...
dev_dependencies:
  build_runner: ^2.0.2
  ferry_generator: ^0.5.0-dev.9
  ...
...

name: console_simple
...
dependencies: 
  ferry: ^0.10.4
  gql_http_link: ^0.4.0
dev_dependencies:
  build_runner: ^2.1.4
  ferry_generator: ^0.4.4
  lints: ^1.0.0

Ok then, let's try my luck with the experimental ferry_generator. The latest version as of writing is 0.5.0-dev.9:

dev_dependencies:
  build_runner: ...
  ferry_generator: ^0.5.0-dev.9
  lint: ...

But the generation fails:

It doesn't even tell me in which file the null exception happened. But the warnings must mean something... Taking a look at the pokemon example, build.yaml (See code), the target names have changed:

With the new build targets in place, the code generation outputs a different file structure, like in the Pokemon example. However, the type mismatch is still here.

Another thing i could try is update analyzer. Cause I also get this warning, which was present before upgrading to ferry 0.5.0-dev:

[WARNING] ferry_generator:serializer_builder on lib/$lib$:
Your current `analyzer` version may not fully support your current SDK version.

Analyzer language version: 2.14.0
SDK language version: 2.16.0

Please update to the latest `analyzer` version (3.0.0) by running
`flutter packages upgrade`.

The flutter packages upgrade runs but does not seem to upgrade analyzer. If I add analyzer to the devDependencies, pub get fails with this error:

Because ferry_generator 0.5.0-dev.9 depends on analyzer ^1.7.2 and no versions of ferry_generator match >0.5.0-dev.9 <0.6.0, ferry_generator ^0.5.0-dev.9 requires analyzer ^1.7.2. So, because console_simple depends on both analyzer ^3.0.0 and ferry_generator ^0.5.0-dev.9, version solving failed.

Now this is weird. The code generation tells me I'm on Analyzer version 2.14.0, but a dart packages upgrade tells me I'm still on version 1.7.2, as ferry_generator requires. Is there something about the environment analyzer version vs the project's installed package version? I'm just guessing here, I don't know much about the Dart or Flutter environment.

My version of Dart seems recent. Does it not ship with a recent version of Analyzer? How do I upgrade Analyzer?

% dart --version     
Dart SDK version: 2.16.0-80.1.beta (beta) (Mon Dec 13 11:59:02 2021 +0100) on "macos_x64"

Another difference is that I'm running Dart, not Flutter, while the Pokemon example relies on Flutter.

I might be shooting myself in the foot by running it in Dart. Ultimately I want to make a graphical dashboard, so I might as well switch to Flutter now. The reason I went for Dart in the beginning, was to start as lightweight as possible to prevent weird compatibility issues. Looks like it brought me issues instead.

Switching from Dart to Flutter

I repeated the same steps and ran the code generator with flutter pub run build_runner build, and the same warning about not using analyzer 3.0.0 came up. The Borrow request also fails to compile because of the String and GStringBuilder type mismatch.

So I used the heaviest weaponry, dependency overrides (see pubspec doc):

dev_dependency_overrides:
  analyzer: ^3.0.0

And code generation did not throw the analyzer version warning this time. However, it still produced the wrong G__typename setter.

How on earth did they generate the code for the Pokemon example?

Looking at pubspec.lock, I can see that it actually didn't install analyzer 3.0.0. It turns out I made up dev_dependency_overrides. There is only one way to override dependencies:

dependency_overrides:
  analyzer: ^3.0.0

And now, the code generation tool crashes:

% flutter pub run build_runner build
Failed to build build_runner:build_runner:
../../../../flutter/.pub-cache/hosted/pub.dartlang.org/dart_style-2.1.1/lib/src/source_visitor.dart:2695:22: Error: Type 'TypeName' not found.
  void visitTypeName(TypeName node) {

So I guess ferry_builder knows what it's doing, and really does not want a newer version of analyzer. The analyzer warning was not the cause of the issue.

Let's just copy/paste the dependencies from the Pokemon example. After trying again, still no luck. Now I'm thinking I should restrict the versions. So I removed a few of the ^ to make the rules stricter.

Now the script only generates two serializers.* files. Btw, the last time the pubspec.yaml was modified was Sept 9.

So one might assume that when that person ran flutter pub get, they got a version more recent than the pubspec.yaml's version of 2.0.2.

  • 2.0.2 produces only two files.
  • 2.1.2 produces the type mismatch. I expect the right version to be somewhere in the middle.
  • 2.0.6 produces only two files.
  • 2.1.0 produces the type mismatch.

So I can't find a version of build_runner that actually produces the right output.

Generating the files in the Pokemon example

Let's see what kind of errors I get if I try to regenerate the Pokemon files.

% flutter pub get
% rm -rf lib/__generated__
% rm -rf lib/src/graphql/__generated__
% flutter pub run build_runner build
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Warning
──────────────────────────────────────────────────────────────────────────────
Your Flutter application is created using an older version of the Android
embedding. It is being deprecated in favor of Android embedding v2. Follow the
steps at

https://flutter.dev/go/android-project-migration

to migrate your project. You may also pass the --ignore-deprecation flag to
ignore this check and continue with the deprecated v1 embedding. However,
the v1 Android embedding will be removed in future versions of Flutter.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
The detected reason was:

  /Users/anhtuan/git/ferry/examples/pokemon_explorer/android/app/src/main/AndroidManifest.xml uses
  `android:name="io.flutter.app.FutterApplication"`
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[INFO] Generating build script...
[INFO] Generating build script completed, took 582ms

[INFO] Initializing inputs
[INFO] Reading cached asset graph...
[INFO] Reading cached asset graph completed, took 80ms

[INFO] Checking for updates since last build...
[INFO] Checking for updates since last build completed, took 606ms

[INFO] Running build...
[INFO] 1.1s elapsed, 4/5 actions completed.
[WARNING] ferry_generator:serializer_builder on lib/$lib$:
Your current `analyzer` version may not fully support your current SDK version.

Analyzer language version: 2.14.0
SDK language version: 2.16.0

Please update to the latest `analyzer` version (3.0.0) by running
`flutter packages upgrade`.

If you are not getting the latest version by running the above command, you
can try adding a constraint like the following to your pubspec to start
diagnosing why you can't get the latest version:

dev_dependencies:
  analyzer: ^3.0.0 

[INFO] 3.0s elapsed, 4/5 actions completed.
[INFO] 4.0s elapsed, 4/5 actions completed.
[INFO] Running build completed, took 4.6s

[INFO] Caching finalized dependency graph...
[INFO] Caching finalized dependency graph completed, took 46ms

[INFO] Succeeded after 4.6s with 26 outputs (16 actions)

Despite the warnings, it generated the code fine, and there is no type mismatch. What is it that I am doing wrong?

Even after simplifying the example query file to:

query AllPokemon {
  pokemons(limit: 5, offset: 0) {
    results {
      height {
        in_meter
      }
    }
  }
}

It still compiles fine.

As a last attempt, I tried to write my own property to the Query type, a simple numbers: [Int] field, and this simple query:

query MyBorrows {
  numbers
}

Still failed.

I also locked the versions to what I found in the Pokemon example's pubspec.lock. No luck.

Then I brought over the Pokemon example's schema.graphql over, and things started to compile again. So it really had to do with the schema after all.

After a bit more poking, it dawned on me that perhaps the schema defines a String type, which interferes with the regular String scalar. I ended up finding this:

It's a bit weird since GraphQL already has those scalars defined by default (see doc).

And after removing this one line, sure enough, it started to compile.

G__typename's setter parameter is finally a String!

So it wasn't a compatibility issue due to Dart or Flutter or analyzer's version nor build_runner's. It wasn't a problem with the stable vs dev version of ferry_generator either.

__typename??

Now we're ready to write the request itself and run the program for the first time:

import 'package:console_simple/__generated__/borrows.req.gql.dart';
import 'package:gql_http_link/gql_http_link.dart';
import 'package:ferry/ferry.dart';

final link =
    HttpLink('https://api.thegraph.com/subgraphs/name/aave/protocol-v2');

final client = Client(
  link: link,
);

void main(List<String> arguments) {
  final borrowsReq = GMyBorrowsReq();
  print(borrowsReq.toJson());
  client.request(borrowsReq).listen((response) {
    print(response.operationRequest.requestId);
    print(response.hasErrors);
    print(response.graphqlErrors);
    print(response.data);
  });
}

This fails with the following error:

[GraphQLError(message: Null value resolved for non-null field `__typename`, locations: [ErrorLocation(line: 2, column: 3)], path: null, extensions: null)]

Meanwhile, borrowsReq.toJson() evaluates to:

{vars: {}, operation: {document: query MyBorrows {
  borrows {
    amount
  }
}, operationName: MyBorrows}, executeOnListen: true}

However, after digging into ferry's Client code, it turns out this is not the actual request made to the GraphQL server. ferry adds the __typename field query to the request root and each requested field. So the request really looks like this:

{
  "operationName":"Borrows",
  "variables":{},
  "query":"query Borrows {\n  __typename\n  borrows {\n    __typename\n    id\n  }\n}"
}

See more references to __typename in the repository: https://github.com/gql-dart/ferry/search?q=__typename.

In the TheGraph playground, I can see that __typename is correctly implemented for the borrows field, but not the Query root:

__typename works fine for returned types
__typename is not implemented on the root Query

So it looks like ferry assumes the root query exposes __typename, but TheGraph subgraphs don't expose it. What does the GraphQL spec say about that? One of them must be wrong, but fixing either would take effort. Honestly I've struggled enough just to read it in Dart. Because then the steps to build my dashboard would basically be:

  1. Run graph-node's example tool to generate a proper GraphQL schema out of protocol-v2-subgraph's schema.
  2. Remove the 400 unnecessary fields from schema.graphql's Reserve_filter so that the code generation tool doesn't crash. (not sure if it's still necessary after removing the String scalar).
  3. Comment out the String scalar.
  4. Generate the code.

Another way would be to use https://pub.dev/packages/graphql. But then it's not typed anymore, and it defeats the purpose of using Flutter in the first place.

At this point it might be more reasonable to use graph-cli to generate the ts files, and make my dashboard in TypeScript. I can use the awesome Vega like I did in http://blog.wafrat.com/graphics-cards-comparison/.