Refresh API

How Refresh Jobs Work

In general, refresh jobs work a bit like search engine crawlers. We request every cached HTML file with an anonymous and fresh session (no cookies etc. will be sent to your origin) from your servers and compare the content. If the content has changed, the whole cache hierarchy is updated.

With growing numbers of files, this procedure takes a bit more time (also because we do not want to overwhelm your server with requests). To speed up the process, you can use a type: "INSTANT" Refresh Job. This means the very first request to an HTML file that has not been refreshed yet, will return the currently cached version and trigger an immediate update in the background. All requests and users after that will get the fresh content from the cache. This ensures that popular sites are updated instantly while maintaining an optimal cache hit rate.

With type: "DEPLOYMENT" you can even force the CDN to be purged hard so no cached content is served afterwards.

Generate API Token

Log in to the Dashboard, go to settings and generate your token under “Refresh API Token”. This token has to be sent with every request.

authorization: BAT <your-token>

Refresh REST Endpoint

All refresh requests are issued against the revalidation endpoint:

https://<appname>.app.baqend.com/v1/asset/revalidate

Starting Refreshes via POST

With a POST request against the endpoint, you can start a new refresh. The POST body should contain the information on what to refresh. On top of that you can set options on how to carry out the refresh. The body is given as JSON:

Syntax

  • Body Syntax

    The body can contain the following attributes

    • filter - A filter to restrict the refreshing assets. The filter can contain the following attributes

      • mediaTypes - Restricting the exact media types to refresh

      • contentTypes - Restricting the content types to refresh. For Example: ["script", "style"]. Possible values are:

        • document

        • style

        • script

        • feed

        • audio

        • video

        • track

        • image

        • font

      • urls - Matching an array of URLs. URLs can include the wildcard character *. For Example: ["<https://example.com/html/example-*>"]

      • prefixes - Matching URLs via given prefixes

      • tags - Matching tags which was previously set with the Baqend-Tags HTTP header when the asset is fetched from the origin

      • query - Containing a MongoDB query to restrict which resources to refresh. This lets you combine attributes with AND and OR or even execute regular expressions. The query can use the following metadata fields:

        • url

        • eTag

        • lastModified

        • contentType

        • mediaType

        • contentHash

        • variation

        • origin

        • tags

    • triggeredBy - Allows to define a name of a routine or person which triggered the job. Default is api

    • allowDelete - Defines whether a refresh job can delete assets if they are unused or gone. Default is true

    • type - Defines how the refresh is carried out. The following values are supported. Default is "REFRESH"

      • "REFRESH" - A standard refresh job that iterates through resources and refreshes them. Resources that are not yet refreshed will still served the cached version.

      • "INSTANT" - The server will not serve cached content until it is refreshed and also do a soft purge against the CDN. This means the first request of a user against the CDN will see a cached version and triggers a refresh in the background, subsequent requests will be fresh. This is a nice trade off to keep cache hit rates high and still update popular resources very fast. Note: should be used to refresh a large amount of HTML files

      • "DEPLOYMENT" - A deployment refresh purges the cache immediately and builds it up again lazy. This means cache entries are immediately up to date. This will cause a performance drop and should only be used if the HTML structure have been changed. (e.g. after a deployment of the origin) Note: only supported for HTML files.

      • "PURGE" - Our caches will receive a complete purge for the marked assets. This will drop all resources from our caches. The cache will not be build up again for the purged resources. Note: this will dramatically impact the performance and should only be used to delete unwanted cached resources from our caches.

    • forceUpdate - Forces to also update resources that are marked as immutable because they contain a hash and have high caching headers. Default is false

    • delay - Defines a wait time between batches of refreshed resources. This option can be used to minimize the traffic to the origin server if necessary but makes the refreshes take much longer. Default is 0

    • scheduleIn - Allows to define a time span in ms the job scheduler will wait until the refresh job is actually scheduled to run. Default is 0

    • name - Set a name for the refresh job to make it easier identifiable. Default is null

    Example:

    {
  "filter": {
    "mediaTypes": [
      "text/plain",
      "video/mpeg"
    ],
    "contentTypes": [
      "image",
      "video"
    ],
    "urls": [
      "<http://localhost:8080/v1/file/www/video>"
    ],
    "prefixes": [
      "<http://localhost:8080/v1/file/www/vi>"
    ],
    "query": {
      "origin": "<http://localhost:8080/v1/file/www/video>",
      "url": { "$regex": "$http://localhost:8080/v1/file/www/video/\\d+" }
    },
  },
  "type": "REFRESH",
  "forceUpdate": false,
  "revalidationDelay": 0
}

  • Response Syntax

    The response of this POST request returns a content-type: application/json;charset=utf-8 containing a statusId that can be used to query the status of the refresh via a GET request to the same endpoint.

    Example Response

    { "statusId" : "bc87f0aa-6013-4d2a-bdc2-9e96c8e86118" }

Example I (refreshing JS and CSS)

{
  "filter": {
    "contentTypes": ["script", "style"],
    "query": { "origin": "https:/www.example.com" }
  },
  "type": "refresh"
}

Refreshes all scripts and styles on the given domain and purges the CDN. Even resources that are considered immutable are refreshed here.

Example II (HTML Refreshing)

{
  "filter": {
    "contentTypes": ["document"],
    "query": { "origin": "<https://www.example.com>" },
  },  
  "type": "instant"
}

Refreshes all HTML files and purges the CDN with a soft purge. This means the very first request to an HTML file that has not been refreshed yet, will return the current cache version and trigger an immediate update in the background. All requests and users after that will get the fresh content from the cache. This ensures that popular sites are updated instantly while maintaining an optimal cache hit rate.

Request as CURL

curl -X POST -d '{ "contentTypes": ["document"], "instant": true, "origin": "https:/www.example.com" }' -H "authorization: BAT <tour-token>" -H "Content-type: application/json" https://<appname>.app.baqend.com/v1/asset/revalidate

Checking Refresh Status via GET

If you need to know the progress of a refresh that you started, you can query it via a GET requests including your statusId like this:

https://<appname>.app.baqend.com/v1/asset/revalidate/<statusId>

  • Response Syntax

    The response contains the following attributes

    • revId - The revalidation ID is equal to the statusId from the request

    • filter - Contains all the details of the refresh that was created via POST

    • assetCount - Contains the number of resource matched by the refresh

    • refreshedCount - Contains how many resources have been updated so far

    • changedCount - Contains how many resources have changed during the update

    • state - Contains the state of the refresh, value are

      • FINISHED - For refreshes that are completed

      • CREATED - The job was created and assets are being marked

      • RUNNING - The marking finished, the job is passed on to the revalidation loop

      • ERROR - An error occurred. The error property will hold more information on the cause.

      • CANCELLED - The job was cancelled either via API or by the job scheduler because another Instant job was running

      • SCHEDULED - The scheduler moved the job aside and waits the time specified in the scheduleIn property

      • DEFERRED - Another job with the same filter is currently active, this job is deferred until the previous job finishes. Subsequent jobs with the same filter as the DEFERRED job are ignored

    • time - The current runtime of the refresh

    • deleteAll -

    • deleteMissing -

    Example

    {
  "revId": "bc87f0aa-6013-4d2a-bdc2-9e96c8e86118",
  "filter": {
    "mediaTypes": [
      "text/plain",
      "video/mpeg"
    ],
    "contentTypes": [
      "image",
      "video"
    ],
    "urls": [
      "<http://localhost:8080/v1/file/www/video*>"
    ],
    "prefixes": [
      "<http://localhost:8080/v1/file/www/vi>"
    ],
    "query": {
      "url": "<http://localhost:8080/v1/file/www/video>"
    }
  },
  "assetCount": 1,
  "refreshedCount": 1,
  "changedCount": 0,
  "final": true,
  "deleteErrors": false,
  "state": "FINISHED",
  "error": null,
  "startedAt": "2021-03-26T07:17:58.295214311Z",
  "updatedAt": "2021-03-26T07:18:25.582033896Z",
  "finishedAt": "2021-03-26T07:18:25.582033896Z",
  "triggeredBy": "api",
  "forceUpdate": false,
  "allowDelete": true,
  "type": "REFRESH",
  "duration": 27287,
  "scheduleIn": null,
  "delay": 0,
  "name": null
}

The Baqend-Tags header

When content is fetched from the origin, a special Baqend-Tags header can be set as well to tag the to be cached resource with some specific tags.

Those tags can be later used in a refresh job, to refresh all resources tagged with a specific value. The tag header should contain tags separated by white spaces, tags can contain any allowed HTTP header chars.

Example:

 Baqend-Tags: some-tag some/other/tag ThirdTag

That resource can be then refreshed with the following Job:

{
  "filter": {
    "tags": ["some-tag", "some-other-resource"]
  },
  "type": "refresh"
}

If you want to use a different header or another separator, then this can be configured by the Baqend staff.

PreviousAccelerating Dynamic DataNextPage Speed Analyzer

Last updated