• If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • You already know Dokkio is an AI-powered assistant to organize & manage your digital files & messages. Very soon, Dokkio will support Outlook as well as One Drive. Check it out today!

View
 

RESTful WebHooks

Page history last edited by nathandw 14 years, 5 months ago

RESTful WebHooks (Recomendation) 

 

Introduction

 

Hey all,

This page is dedicated to the development of standards/guidelines for RESTful WebHooks. This page should be used to throw around ideas about how it ought to be done. As these ideas become more structured and coalesce so may this page. I am no Wiki master so if you would like to define a more coherent structure please do. Please post any thoughts you may have in the comments section below, this will serve as a record for the reasoning behind changes in this document. Please avoid making significant changes to this document without first posting suggestions for "pier review." If you want to provide examples of your proposals please link to publicly accessible external documents, lets try to avoid cluttering the comments section with examples.

 

Note: This is still a very rough concept as such you should expect that the recommendations on this page will change. This document is also being maintained on Google Wave, if you have a membership and would like to participate let us know.

 

What are RESTful WebHooks

Briefly, some history and bearings to lay the groundwork for the definition. REST was first "fielded" by Roy Fieldings, in chapter 5 of his doctoral dissertation. See a more readable write-up at RestWiki. REST is, at its core, a sensible way to think about and implement your URIs (the resources - nouns) and HTTP (the interaction with the URIs - verbs). REST reveals that all web sites/applications are web services and the HTTP supports all that is necissary for web services.REST treats a URI as a unique identifier for a resource whether that resource is an image, web page, or complex custom application object.

 

RESTful WebHooks support subscription, notification, and publication through a RESTful interface. This means that HTTP Verbs GET, POST, PUT and DELETE must act on subscription, notification, and publication resources according to REST definitions. RESTful WebHooks MAY overload the POST verb where necessary to support incompatible clients, however, support for native verbs MUST be available. RESTful WebHooks MUST NOT overload the GET verb except where custom idempotent and safe verbs have been defined. RESTful WebHooks MUST support four event types: ACCESSED, CREATED, UPDATED and DELETED. These event types correspond to the four HTTP verbs and notifications should be dispatched when these actions are applied to a resource. If custom HTTP verbs are implemented in your REST design you may add additional event types correlating to the additional verbs (this is generally discouraged). While all HTTP verbs and events must be supported this does not mean that all resources must allow them. You MAY have resources that do not allow the DELETE command, and so a DELETED notification MAY never be dispatched for this resource. Likewise, subscription resources MAY restrict access to verbs such as DELETE.

 

The only restriction on the Body of a request is that data held within the Body of a request MUST NOT contradict WebHooks values in the HEADER of the request; for example, if the HEAD specifies a Notification-Type of UPDATED the Body MUST NOT contradict this type, if it does the type in the HEAD MUST be respected. RESTful WebHooks make allowance for REST+RPC hybrids, the URIs for subscription, publication and notification are not required to be RESTful as long as the HTTP Verbs act on these URIs in the expected manor. While not recommended the Body of a request may also contain un-RESTful data such as SOAP.

 

Subscription, notification and publication entities MUST all be treated as RESTful resources. This means that POSTing to a subscription URI MUST create a subscription resource. DELETEing a subscription resource MUST constitute an un-subscription. While a RESTful WebHooks SubPubHub is not required to permit GET, PUT, or even DELETE these must be the mechanisms through which subscription resources are interacted with. This means you MUST NOT provide this functionality through other means; DELETE MUST be the only mechanism through which to unsubscribe from a subscription.

 

RESTful WebHooks does not define standard formats for subscription, notification, and publication resources or payloads. Resource formats may vary based on the support and features that individual SubPubHubs offer. While standards for these entities are desirable we will omit them from this specification for the sake of flexibility. This will allow implementors of RESTful WebHooks to subscribe to resource and payload standards that best fit their particular use case.

 

Discovery

WebHooks subscription services are discovered by issuing a HEAD request to the resource URI. The URI of the subscription resource MUST be returned in a Link header marked "rel=Subscriptions". The HEAD request MAY return any number of additional headers or body. In this example the Allow header is returned specifying the content negotiation options. The subscription URI SHOULD be constructed in a RESTful manor as subscriptions MUST be treated as RESTful resources.

 

=> HEAD http://www.webapp/resources/resource/abcd

 

<=

200 Ok

Allow : application/json, application/xml

Link : <http://www.webapp.com/resources/resource/abcd/web-hooks>, rel="Subscriptions"

 

Subscription

To subscribe to a WebHook you MUST POST the subscription data as well as the desired Notification-Type to the subscription URI. The response MUST contain the location of the subscription resources as well as a Link to the Subscriptions resource. The body of the subscription POST in this example is a simple URI, RESTful WebHooks make no assertions involving payload content however the Body of the POST command should usually contain subscription data from which a new subscription resource is created. This means the Body could contain hub specific subscription data, allowing WebHooks to be published by SubPubHub hosts with a variety of subscription options.

 

Proprietary Hub Subscription (Inlined)

=>

POST http://www.webapp.com/resources/resource/abcd/web-hooks

Content-Type : text/uri-list

Notification-Type : UPDATED

 

http://www.webscripts.com/myaccount/mylistener

 

<=

201 Created

Location : http://www.webapp.com/resources/resource/abcd/web-hooks/web-hook/1234

Link : <http://www.webapp.com/resources/resource/abcd/web-hooks>, rel="Subscriptions"

 

Public Hub Subscription

=>

POST http://www.subpubhub.com/myaccount/www.webapp.com/resources/resource/abcd/web-hooks

Content-Type : text/uri-list

Notification-Type : UPDATED

 

http://www.webscripts.com/myaccount/mylistener

 

<=

201 Created

Location : http://www.subpubhub.com/myaccount/www.webapp.com/resources/resource/abcd/web-hooks/web-hook/1234

Link : <http://www.subpubhub.com/myaccount/www.webapp.com/resources/resource/abcd/web-hooks>, rel="Subscriptions"

Link : <http://www.webapp.com/resources/resource/abcd>, rev="Resource"

 

PubSubHubbub Subscription (Response Not RESTful)

=>

POST http://myhub.example.com/

Content-Type : application/x-www-form-urlencoded

Notification-Type : UPDATED

 

hub.callback=http://www.webscripts.com/myaccount/mylistener

&hub.mode=subscribe

&hub.topic=http://www.syndicate.com/topic

&hub.verify=sync

 

<=

204 No Content

 

Publication

In order to publish a notification to a SubPubHub the publisher MUST POST notification data to the WebHooks notification queue for the target resource. The publisher MUST include a Link to the Resource that has been acted upon as well as a Notification-Type. The publisher can optionally include an ACKnowledgment Link to which the SubPubHub will post acknowledgements. The body of the post may contain content, WebHooks is payload agnostic and so any custom notification resource may be posted. The SubHubPub MUST return the Location of the newly created notification resource. This notification resource can be queried for statistics involving notification status. The notification can also be deleted preventing any unsent messages from being sent. In the first example a simple custom notification json object is passed setting the number of retires for this notification.

 

Resource Publishes Notification to SubPubHub

=>

POST http://www.webapp.com/resources/resouce/abcd/web-hooks/notifications

Content-Type : application/json

Notification-Type : UPDATED

Link : <http://www.webapp.com/resources/resource/abcd>, rel="Resource"

 

{ retries : 10 }

 

<=

201 Created

Location : http://www.webapp.com/resources/resouce/abcd/web-hooks/notifications/notification/1324

 

 

Resource Publishes Notification to PubSubHubbub (Response Not RESTful)

=>

POST http://myhub.example.com/

Content-Type : application/x-www-form-urlencoded

Notification-Type : UPDATED

Link : <http://www.syndicate.com/topic>, rel="Resource"

 

hub.mode=publish

&hub.url=http://www.syndicate.com/topic

 

<=

204 No Content

 

 

A RESTful SubPubHub MUST distribute messages to the subscribers passing any custom headers to the subscriber. The SubPubHub MAY attach additional headers for the various Links of interest. The SubPubHub MAY also attach a ACKnowledgemnt Link which can forward responses to the Link supplied by the notifier (if any). The SubPubHub SHOULD not include the resource in the payload; the subscriber uses the Resource Link to access the resource, this improves performance, ensures resource integrity, and prevents the PubSubHub from having to manage content negotiation. WebHooks is payload agnostic allowing the SubPubHub to include any hub specific payload for the publication, this means the SubPubHub MAY include the resource if so desired.

 

SubPubHub Distributes Message to Subscribers

=>

POST http://www.webscripts.com/myaccount/mylistener

Notification-Type : UPDATED

Link : <http://www.webapp.com/resources/resource/abcd>, rel="Resource"

Link : <http://www.webapp.com/resources/resource/abcd/web-hooks>, rel="Subscriptions"

Link : <http://www.webapp.com/resources/resource/abcd/web-hooks/web-hook/1234>, rel="Subscription"

Link : <http://www.webapp.com/resources/resource/abcd/web-hooks/notifications/notification/1234/acknowledgments/acknowledgment/1234, rel="ACK"

 

<=

200 Ok

 

Acknowledgement

A notification MAY include an ACKnowledgement Link if acknowledgement is expected. POSTing to the acknowledgement link MAY notify the SubPubHub that the message was received removing it from the notification queue. Acknowledgement MUST be forwarded to the notification source if the ACKnowledgement Link was POSTed as part of notification creation.

 

=>

POST http://www.webapp.com/resources/resource/abcd/web-hooks/notifications/notification/1234/acknowledgments/acknowledgment/1234

 

 

 

Comments (14)

Daniel Parker said

at 6:41 pm on Oct 6, 2009

How / where should we (I and others) discuss this critique the content on this page?

nathandw said

at 6:50 pm on Oct 6, 2009

I would suggest posting comments here, in the comments section. If it is a controversial topic we can argue back and forth until we reach a consensus (or kill each other :). We can then amend the document to reflect the new ideas. This has the added benefit of documenting the reasons for changes in the document. If it is a major controversy perhaps we can even create a new page just for discussing the topic.

Daniel Parker said

at 10:13 pm on Oct 6, 2009

I would like to NOT suggest the use of a relative URL simply to use for a HEAD request with web-hook headers. This document says "It also prevents us from having to define special URLs..." but then it does define a special URL, albeit relative to a given resource. First of all, I don't want any web standard to mandate what a URL should be named. What if I want to have a separate resource named "web-hook"? Secondly, in following with the fundamental principles of REST as put forth by Roy Fielding (the originator of REST), a URL is to be thought of as a handle to a resource (http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm#sec_5_2_1_1). I am proposing that a "subscribe" request should not be thought of as creating a new *resource* (a "subscription" resource) but rather an *action* that can be performed on an existing resource. (In fact, as a side note, it could be that we should seek to add SUBSCRIBE as a verb alongside GET and HEAD, with its own specific purpose, but good luck with that!) If a subscription should not be considered a resource, then it would not be in keeping with fundamental REST principles to use a different URL to represent it. In a sense, this document is proposing that a HEAD request for /web-hooks would, instead of communicating information about the /web-hooks "resource", it would communicate information about the capabilities and options of a different URL, the resource at a relatively-derived URL.

Simple HTTP headers on the resource itself are sufficient, and should actually be returned by a HEAD request on the resource URL itself.

Daniel Parker said

at 10:14 pm on Oct 6, 2009

Now, we all know that it may be difficult in some situations to add the ability to subscribe, to each and every resource at its home URL. And of course, many will even want to (or need to) outsource the subscription-fulfillment to another server -- or even another vendor/domain. In any possible situation, it would work to add one more header, "X-WebHooks-Subscription-Location", with the value being a fully-qualified URL that could be thought of as the subscription handler for this resource. A request for subscription must include a parameter naming the resource being subscribed. That way we can all implement a subscription-handling service however we want to -- all resources serviced by the same URL on the same or different server, or using a relative url such as /web-hooks if desired.

Daniel Parker said

at 10:38 pm on Oct 6, 2009

Interesting, Roy Fielding of REST has been long working on a new protocol to replace HTTP: Waka.
> http://en.wikipedia.org/wiki/Waka_(protocol)
> http://gbiv.com/protocols/waka/200211_fielding_apachecon.ppt
In the powerpoint link, slide 16, is reference to a "MONITOR" verb, which is used to watch for changes to a resource. Just an interesting thing to see that the REST guy is seeing the need for more verbs, specifically this one.

Also, the powerpoint has some really good points on making a new protocol (which is what this RESTful WebHooks document is).

nathandw said

at 9:31 am on Oct 7, 2009

You have some good points Daniel, and I like the idea of integrating WebHooks directly in the HTTP protocol. Here are my concerns:
1. New verbs == no support. There are likely clients and REST frameworks that are badly designed and do not have a mechanism for supporting additional HTTP Verbs. Case in point, HTML4/5; we are just now getting PUT and DELETE support in HTML 5, HTML 5 does not include WebDav Verbs so what are the chances of WebHooks Verbs being in HTML 6? Probably None.
2. Wont the MONITOR verb require *POST*MONITOR, *PUT*MONITOR, *GET*MONITOR and *DELETE*MONITOR? The only other option I can think of is to include a header that defines the action, which is not very RESTful.
3. If you perform WebHooks operations on the target resource using standard methods (POST,PUT,GET,DELETE) your are overloading each method. For example PUT http://../resource with an X-WebHooks-... results in subscription update, however, PUT http://.../resource without an X-WebHook-... header results in an update to the resource. Overloaded verbs are frowned on.

I think adding an X-WebHooks-Location header directly to the resource is a good idea, as you pointed out you may want both a web-hooks resource as well as the WebHooks subscription resource subordinate to the same parent resource. Because of issues 1, 2 and 3 I think treating web-hooks as resources is still the best approach for a RESTful design. Particularly if you "outsource" the subscription/publication process; if web-hooks are treated as resources this outsourcing is easy, simply access the resource on the host for sub/pub etc. If you instead treat WebHooks as an HTTP Verb you cannot easily "outsource" the sub/pub. If you have an example of how the subscription process might work via MONITOR please include a link to a public document illustrating this.

Any chance there is a pun intended with WAKA? Waka, Waka (Fozzie Bear; bad joke but couldn't resist)

Great comments,
Nathan

Daniel Parker said

at 11:14 am on Oct 7, 2009

Here I have some examples of what I'm thinking: https://gist.github.com/780ccb1fa8943700515a

nathandw said

at 11:34 am on Oct 7, 2009

Ok, I like what I am seeing there Daniel. If I am reading correctly we treat subscriptions as REST resources, but we locate the subscription resource using the X-WebHooks-Location header for the target resource instead of mandating URL conventions. This allows us to easily "outsource" the pub/sub to a hub (pubsubhubub :) and does not require us to create any additional HTTP Verbs.

On a side note one of the big issues with pub/sub is message delivery, systems like MSMQ ensure that messages get delivered even if the target is offline for a period. It would be great to have an open pub/sub server that could handle all those details for us, ideally it would act as a proxy simply passing the posted event through to multiple targets so that there is no need for a custom package that the pub hub interprets.The pub hub would simply read the X-WebHooks-Location header and rePOST the exact HTTP message to each subscriber. The event source could POST a multi-part message containing payloads corresponding to the X-WebHooks-Allow types associated with the event. The pub hub could then split the multi-part message and send each subscriber the version they requested via X-WebHooks-Accept.

Daniel Parker said

at 11:44 am on Oct 7, 2009

> It would be great to have an open pub/sub server that could handle all those details for us, ideally it would act as a proxy simply passing the posted event through to multiple targets so that there is no need for a custom package that the pub hub interprets.

Yes, I've thought of this many times. You could use it as a web-hook consumer, by creating a webhook on the proxy, then hooking that proxy to the hookable resource; or you could use it as a web-hook provider, by creating a hook point on the proxy, subscribed to all events and all data types, and set your X-WebHooks-Location header to the appropriate url on the proxy.

Also, you could make your own webhook server (or install and configure an opensource webhook server) that sits on the same web server and responds to data changes, or receives event notifications from the app and sends out payloads. Or a separate server on the same LAN. This setup makes it very versatile to however you want to manage.

It really comes down to thinking of the resource and the webhook being separate but related resources. Associate the two resources via the "foreign key" called X-WebHooks-Location. A Resource has-one WebHook resource :)

nathandw said

at 4:09 pm on Oct 7, 2009

The REST-* initiative is developing a messaging standard that we should be looking at:
http://www.jboss.org/reststar/specifications/messaging.html I feel like we are addressing different sides of the same coin. REST-* appeals to developers who are familiar with WS-* while WebHooks could appeal more to casual developers and hopefully the general public. WebHooks could be used to brand REST-Messaging making it palatable to the masses. But this requires that REST-Messaging not be a heavy handed dev-head protocol.

nathandw said

at 4:25 pm on Oct 7, 2009

If I have 1,000,000 subscribers is it more performant to send them all a message containing the full payload or is it better to send them a notification that links to a cacheable payload?

nathandw said

at 7:33 am on Oct 8, 2009

I updated the document to reflect the discovery mechanism we have been discussing. I think we should also consider adopting some of the ideas from the REST-* group.

nathandw said

at 7:38 am on Oct 8, 2009

Should we repalce the "X-WebHooks-Location: ..." with "Link: <http://www.subpubhub/accounts/...>, rel='WebHooks'" or go even more generic "<http://www.subpubhub/accounts/...>, rel='Subscription'"? This makes parsing the URL slightly more complicated but on the other hand is more standards compliant.

nathandw said

at 5:32 pm on Oct 8, 2009

I think in an effort to be RESTful we should try to replace custom "X-WebHooks-..." headers with standard or at least evolving HTTP headers wherever possible. For this reason I replaced X-WebHooks-Location/Id/Help with HTTP "Link:" headers. Also, I am having a really hard time getting the built in editor to work correctly so the formatting on some of the request/response examples is messed up.

You don't have permission to comment on this page.