Cool. CoolQLCool

CoolQLCool (CQC) is an open source GraphQL server that allows you to turn websites into GraphQL APIs. It's pretty tubular (I'm incredibly biased tho).

You can play around with it in GraphiQL. Or take a gander at the source on GitHub.

Example

Features

In order of how radical I think they are.

• Waits for JS to do stuff if you specify.

  By default CQC will load just the HTML like a normal AJAX request, but if you pass the `wait` or `waitForSelector` arguments in your initial `site` search, CQC will wait a designated time, or for a CSS selector on the page to show up.

• Allows you to query multiple sites at once

  In past renditions of web scrapers they've only been able to scrape a single site at once (e.g. Jam API). With CQC you can query multiple sites in a single request by aliasing each `site` call.

• Easier to add more fields in the future.

  With past scrapers it was incredibly difficult to add new fields to query elements by or to grab off of elements. With CQC it will be incredibly easy to have contributors add new fields and functionality.

Possible Features

Some possible features that I'd like to see CQC have some day

• Provide steps to do on a page when it loads
• Recursive fields, for example for pagination on Hacker News to go to a certain number of pages and query each one for the same fields.
• Field to return the output of JavaScript

Examples

curl -X GET -H "Content-Type: application/json" -d '{"query": "{   site(url: \"https://news.ycombinator.com\") {     titles: selectAll(elem: \"tr.athing\") {       id: attr(name: \"id\")       numberOfLinks: count(elem: \".storylink\")       link: selectAll(elem: \".storylink\") {         text         href         class         classList       }     }   } }"}' https://coolql.cool/graphql

var xhr = new XMLHttpRequest();
xhr.responseType = 'json';
xhr.open('POST', 'https://coolql.cool/graphql');
xhr.setRequestHeader('Content-Type', 'application/json');
xhr.setRequestHeader('Accept', 'application/json');
xhr.onload = function () {
  console.log('data returned:', xhr.response);
}
xhr.send(JSON.stringify({ query: `{
  site(url: "https://news.ycombinator.com") {
    titles: selectAll(elem: "tr.athing") {
      id: attr(name: "id")
      numberOfLinks: count(elem: ".storylink")
      link: selectAll(elem: ".storylink") {
        text
        href
        class
        classList
      }
    }
  }
}` }));

require 'net/http'
require 'uri'
require 'json'

uri = URI.parse("https://coolql.cool/graphql")

header = {'Content-Type': 'application/json'}
query = { query: '{
  site(url: "https://news.ycombinator.com") {
    titles: selectAll(elem: "tr.athing") {
      id: attr(name: "id")
      numberOfLinks: count(elem: ".storylink")
      link: selectAll(elem: ".storylink") {
        text
        href
        class
        classList
      }
    }
  }
}' }

http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new(uri.request_uri, header)
request.body = query.to_json

response = http.request(request)