DEPRECATION NOTICE:
rest-feeds is deprecated and will not be further developed. Use http-feeds instead.

Asynchronous data replication and event streaming with plain REST/HTTP.

A REST feed provides access to resources in a chronological sequence of changes.

GET /movies
Accept: application/json

200 OK
Content-Type: application/cloudevents-batch+json
[{
  "specversion" : "1.0",
  "type" : "org.example.movie",
  "source" : "/movies",
  "id": "11b592ae-490f-4c07-a174-04db33c2df70",
  "time": "2019-12-16T08:41:519Z",
  "subject": "18",
  "data": {
    "original_title":"The Fifth Element",
    "popularity":26.163
  }
},{
  "specversion" : "1.0",
  "type" : "org.example.movie",
  "source" : "/movies",
  "id": "64e11a7a-0e40-426c-8d81-259d6f6ab74e",
  "time": "2019-12-16T09:12:421Z",
  "subject": "12",
  "data": {
    "original_title":"Finding Nemo",
    "popularity":23.675
  }
}]

The number of items returned by the server is limited, e. g. to 1000 items per request. Repeat the request with the id that has been processed as lastEventId query parameter. Note that this is an example of an Aggregate Feed where resources can be updated and deleted:

GET /movies?lastEventId=64e11a7a-0e40-426c-8d81-259d6f6ab74e
Accept: application/json

200 OK
Content-Type: application/cloudevents-batch+json
[{
  "specversion" : "1.0",
  "type" : "org.example.movie",
  "source" : "/movies",
  "id": "756e21c9-4ebd-4354-8f7d-85cd7d2bc4ec",
  "time": "2019-12-17T11:09:122Z",
  "subject": "18",
  "data": {
    "original_title":"The Fifth Element",
    "popularity":27.011
  }
},{
  "specversion" : "1.0",
  "type" : "org.example.movie",
  "source" : "/movies",
  "id": "e510d24e-bf06-4f6a-b6db-5744f6ff2591",
  "subject": "12",
  "time": "2019-12-18T17:00:786Z",
  "method": "DELETE"
}]

The server implements long polling. When there are no newer items, the server holds the connection open until new data or a timeout arrives.

Example for no new data after 5 seconds:

GET /movies?lastEventId=e510d24e-bf06-4f6a-b6db-5744f6ff2591
Accept: application/json

200 OK
Content-Type: application/cloudevents-batch+json
[]

The client continues polling this link until new items are received.

REST feeds provide access to resources in a chronological sequence of changes using plain HTTP(S).

A REST feed complies to these principles:

• HTTP(S) as transfer protocol
• Clients poll the feed endpoint for new items
• Batched results with lastEventId to scroll through further items
• Content-Negotiation, with application/cloudevents-batch+json as default
• Feed items are immutable.
• Items are always appended.

REST feeds enable asynchronously decoupled systems without shared infrastructure.

REST feeds can be used for data replication (aggregate feeds) and event streaming (event feeds).

Aggregate feeds are used to share aggregates (aka master data, domain objects) with other systems for data replication and to trigger actions.

Every update to the resource leads to a new entry of the full current state in the feed.

An aggregate feeds must contain every resource (identified through its subject) at least once.

Event feeds are used to publish domain events that have happened in a domain. Event feeds often contain different entry types.

Events may be deleted once they are outdated or useless.

The feed endpoint must return items in a strictly ascending order of addition to the feed.

The feed endpoint must support fetching items without any query parameters, starting with the first items.

The feed endpoint may choose to limit the number of items returned in a response to a subset of the whole set available.

The feed endpoint must must support the lastEventId query parameter. It must only returns items after this event id.

The feed endpoint must implement long polling. If the server has no newer items, the server holds the request open until new items arrive and then immediately sends the response.

The server should send an empty response if no new item arrived after N seconds (5 seconds recommended).

The initially stored link is the feed endpoint URL without query parameters.

Pseudocode:

link = "https://feed.example.com/movies"
while true:
  try:
    response = GET link
    for item in response:
      process item
      link = link + setQueryParameter "lastEventId" to value item.id
  except:
    wait N seconds

The client must persist the id of the last processed item.

The client's item handling must be idempotent (at-least-once delivery semantic). The id may be used for idempotency checks.

The client must implement an exception handler that delays the next request, to protect the server in case of connection or processing errors.

The response contains an array of items. The response must comply with the CloudEvents Specification.

Further metadata may be added, e.g. for traceability.

Every consumer and provider must support the media type application/cloudevents-batch+json as defined by the CloudEvents JSON Batch Format. It is the default and used when the Accept header is missing, defined as plain application/json or not supported.

Further media types may be used, when supported by both, client and server:

• text/html to render feed in browser
• application/atom+xml to support feed readers
• text/event-stream for server-sent events
• application/x-ndjson to support streaming
• multipart/* to have a more HTTP native representation
• application/x-protobuf to minimize traffic
• any other

Feed endpoints may be protected with HTTP authentication.

The most common authentication schemes are Basic and Bearer.

The server may filter feed items based on the principal. When filtering is applied, caching may be unfeasible.

When feed items include the full current state of the resource, older feed items for the same resource may be obsolete. Items may be deleted from the feed when another item was added to the feed with the same resource URI.

It is good practice to keep the feed small to enable a quick synchronization of new clients.

The server must handle next links, when the requested item has been deleted by returning the next higher items.

When a resource was deleted, the server must append a DELETE item with the resource URI to delete.

Clients must delete this resource or otherwise handle the removal.

{
  "specversion" : "1.0",
  "type" : "org.example.movie",
  "source" : "/movies",
  "id": "8a22af5e-d9ea-4e7f-a907-b5e8687800fd",
  "time": "2019-12-17T07:07:777Z",
  "subject": "777777",
  "method": "DELETE"
}

The server should start a compaction run afterwards to delete previous items for the same resource.

Feed endpoints may set appropriate response headers, such as Cache-Control: public, max-age=31536000, when a page is full and will not be modified anymore.

• restfeed-server-java
• restfeed-server-spring
• restfeed-server-spring-example
• restfeed-client-java
• restfeed-client-spring

2021-11-29: Comply with CloudEvents

• Data Model: Add field specversion
• Data Model: Remove field next. Specified query parameter lastEventId instead.
• Data Model: Rename timestamp to time
• Data Model: Rename resource to subject
• Data Model: type should be reverse-DNS name.
• Media Type: Respond with application/cloudevents-batch+json

• FAQ
• Design decisions