afHttpClientExt

HTTP Client provides powerful Axon functions for making HTTP requests and calling REST APIs.

HTTP Client is FREE to download and evaluate.

HTTP Client is a complete custom re-write of the HTTP protocol, wrapped up in a handful of simple Axon functions. HTTP Client boasts the following features:

• support for all HTTP methods, GET, POST, PUT, DELETE, OPTIONS, etc...
• support for custom HTTP headers
• post forms, JSON, and XML
• upload files and multi-part forms
• BASIC, BEARER, DIGEST, Project Haystack / SkySpark SCRAM, header, path, and query authentication mechanisms
• gzip and deflate response de-compression
• redirect following
• session cookies
• JSON pretty printing

HTTP Client Ext v2 has a clean and simple API for building REST requests, it maintains session state including cookies, and has built in extensible support for authentication schemes.

• Axon Functions
• Body sniffing
  • XML / JSON / Plain Text
  • Zinc
  • Form
  • File
  • Multipart Form
• Authentication
  • Keep passwords safe
  • OAuth2 / Custom Authentication
    • Debug Logs
• HTTP Errors
• Debugging
• SSL Handshake Exceptions
• Licensing
  • Evaluation Mode
• Disclaimer
• Special Mentions

HTTP Client is a SkySpark extension requiring SkySpark v3.1 or later. For compatibility with SkySpark v3.0, download HTTP Client v2.0.

HTTP Client Ext is free to download and evaluate but a licence must be purchased for production use. See Licensing for details.

Need help!? Are you confused by cloud connectors? Feeling jaded by JSON? Or just adverse to authentication APIs?

Then let Fantom Factory help! We create mini SkySpark extensions that add scalable and secure authentication to HTTP Client for any cloud platform. Speak to us to find out more.

Axon Functions

HTTP Client Ext provides the following Axon functions:

• afHttpGet - send a HTTP GET request
• afHttpPost - send a HTTP POST request
• afHttpRequest - send a HTTP request with custom method
• afHttpResponse - returns details on the last HTTP response
• afHttpAuth - authenticates with the end server
• afHttpCookies - returns a Grid of cookies (yum!)
• afHttpConfig - sets config for the current session
• afHttpClear - clears the current settings / session
• afHttpDebug - turns request / response debugging on / off
• afHttpReadFromJson - reads Axon objects from standard (non-Haystack) JSON strings
• afHttpWriteToJson - writes (and pretty prints) Axon objects to standard (non-Haystack) JSON strings
• afHttpUriPlusQuery - adds query parameters to Axon URIs

The simplest function to retrieve the contents of a URL would be:

afHttpGet(`https://www.fantomfactory.com/`)

Beyond this, HTTP Client Ext maintains the notion of a session for the duration of the Axon context it runs it. (The scope of the Axon context is usually the top-level function being run.) During this session, cookies are saved and automatically resent, as are any authentication credentials.

Because HTTP Client is designed to be used inside Axon functions, it can be problematic when making HTTP calls in the standard Tools App - because in the Tools app, the session ends after each command.

To get around this, wrap your calls to HTTP Client Ext in a mini-function. For example, to return the HTTP response for display in the tools app, you could call:

(()=>do afHttpGet(`https://www.fantomfactory.com/`); afHttpResponse({camelCase}); end)()

Body sniffing

HTTP Client Ext makes it easy to send different types of content by passing different objects as the body argument to HTTP functions.

The body is inspected (sniffed) and handled appropriately, as outlined below. HTTP Client Ext takes care of all the intricate technical stuff, like setting the correct HTTP request headers, so you don't have to.

Str to JSON / XML / Plain Text

If body is a Str then it is sent (as UTF-8 encoded characters):

If the string starts with < and ends with > then it is taken to be XML and the content type set to text/xml.

If the string starts with { and ends with }, or if it starts with [ and ends with ] then it is taken to be JSON and the content type set to application/json.

All other strings are taken to be plain text and the content type set to text/plain.

afHttpPost(`http://example.com`, null, """{"name":"Steve", "beer":"London Pride"}""")

Grid to Zinc

If the body is a Grid then it is written out in zinc format and the content type set to text/zinc.

afHttpPost(`http://example.com`, null, readAll(equip))

Dict to Form

If the body is a Dict then it is taken to be form values and written out appropriately. The content type is set to application/x-www-form-urlencoded.

This mimics submitting a form on a web page.

afHttpPost(`http://example.com`, null, {name:"Emma", score:11})

Uri to File

If the body is a Uri and starts with io/ then it is taken to represent a file in the project's io/ directory. The file is then read and its binary content piped out as the request body. The file's extension is used to set the content type.

Note that most file uploads are performed via multipart-forms, so try that before this.

afHttpPost(`http://example.com`, null, `io/upload.zip`)

Fn to Multipart Form

Axon Funcs are used to send multipart forms, which are usually used to upload files.

afHttpPost(`http://example.com`, null) (addPart) => do
    addPart("name", "Steve")
    addPart("file", `io/upload.zip`)
end

addPart() takes a name and a value, where name is the name of the form entry, and value may be any Grid, Str, or Uri object which is sniffed just like the body object.

REST APIs

The afHttpReadFromJson() and afHttpWriteToJson() functions can be used to send / receive JSON from REST APIs.

// a sample object structure
dict : {name:"Emma", sex:"female", score:11, likes:["Cakes","Adventure"]}

// convert the object to a pretty JSON string
json : dict.afHttpWriteToJson({prettyPrint, maxWidth:20})

// most REST APIs require some kind of authentication
afHttpAuth("bearer", "XXXX-XXXX")

// make the call
res  : afHttpPost(`http://example.com/`, null, json)

// convert the returned JSON string back into Axon objects
// use 'camelCase' if you want to display data in the Tools app
data : res.afHttpReadFromJson({camelCase})

Don't forget that PUT, DELETE, OPTIONS, and any other HTTP method may be invoked using the more generic afHttpRequest() function.

Authentication

HTTP Client Ext comes with default implementations of BASIC, BEARER, DIGEST, Project Haystack SCRAM, and custom header, query, path, and cert authentication schemes. To use them, call afHttpAuth() at the start of your session and the authentication mechanism will be played out on the next HTTP request.

Authentication mechanisms cache their values and re-use them in calls to the same domain, so they only need to be set the once at the start of the current session.

afHttpAuth("basic", "user", "secret")        // set auth credentials
afHttpGet(`http://example.com/secret1.txt`)  // authentication happens here
afHttpGet(`http://example.com/secret2.txt`)  // cached auth values are re-sent

afHttpGet(`http://fantom.org/doc/`)          // auth values are NOT sent to different domains

Valid / supported authentication schemes to pass to afHttpAuth():

• basic - HTTP Basic Authentication as per RFC7617
• bearer - OAuth 2.0 BEARER token as per RFC6750 (set username to the token)
• digest - HTTP Digest Access Authentication as per RFC7616
• haystack - SCRAM over SASL as per Project Haystack Authentication
• header - Sets a custom HTTP request header to the given value
• path - Sets a named URL path segment to the given value
• query - Sets a URL query parameter to the given value
• cert - Authenticates via an SSL certificate

While basic, bearer, digest, and haystack are Internet standards, header, path, query, and cert are custom schemes that address common authentication scenarios.

header sets a custom HTTP request header value.

afHttpAuth("header", "X-Auth", "xxxx-xxxx")
afHttpGet(`http://example.com/api/endPoint`)

GET /api/endPoint HTTP/1.1
Host: example.com
X-Auth: xxxx-xxxx

path replaces a named URL path segment with the given value.

afHttpAuth("path", "<api-key>", "xxxx-xxxx")
afHttpGet(`http://example.com/api/<api-key>/endPoint`)

GET /api/xxxx-xxxx/endPoint HTTP/1.1
Host: example.com

query sets a URL query parameter with the given value.

afHttpAuth("query", "api-key", "xxxx-xxxx")
afHttpGet(`http://example.com/api/endPoint`)

GET /api/endPoint?api-key=xxxx-xxxx HTTP/1.1
Host: example.com

cert sets the underlying client's socket config with the keystore from the specified certificate.

The username MUST be a Uri (SkySpark IO handle) that resolves to the certificate file. Certificate file handles are resolved in the same format as ioHandle. Certificates must be in a Base64 PEM encoded format, such as .cer, .crt, or .p12.

The password is the passphrase used to unlock the certificate. If left as null, then both null and an empty string "" are attempted to unlock the certificate.

afHttpAuth("cert", `io/certs/certificate.p12`, "xxxx-xxxx")
afHttpGet(`http://example.com/api/endPoint`)

These HTTP Client authentication mechanisms become very secure when used in conjunction with SkySpark's secure store, especially as credentials are automatically removed from any debug logs. See below for details.

Keep passwords safe

Be safe, do NOT store passwords in code or plain text!

Use the core Axon function passwordSet() to set your password in SkySpark's secure store. passwordSet() may be called manually at any point in time using a known Ref.

passwordSet(@xxxx-xxxx, "secretPassword")

Then, in your HTTP code, pass the same Ref to afHttpAuth() and the password will be retrieved from the SkySpark secure store.

afHttpAuth("basic", "user", @xxxx-xxxx)
afHttpGet(`http://example.com/secret.txt`)

This keeps secrets safe from prying eyes!

This works for all custom header, path, and query authentication schemes and Bearer tokens too.

afHttpAuth("bearer", @xxxx-xxxx)
afHttpGet(`http://example.com/secret.txt`)

Debug Logs

All passwords and credentials set via afHttpAuth() will be masked (starred) out in debug logs, keeping them safe.

HTTP Request to: https://example.com
GET /api/********/someEndPoint?q=******** HTTP/1.1
Host: example.com
Authorization: ********
X-Auth: ********
User-Agent: afHttpClient/2.1.6 (Fantom/1.0.77.3103 win32-x86_64)

OAuth2 / Custom Authentication

Many REST APIs have custom authentication schemes and protocols. Most of these are possible to code yourself in Axon using HTTP Client Ext, but doing so leaves you in the sticky situation of having to look after your passwords in plain text.

Also, not every authentication mechanism is straightforward. OAuth2 in particular can be very complex requiring the storage of long term tokens, is largely non-standard, and is more of a general approach to authentication than an actual implementation standard - which is why HTTP Client does not and can not nativly support OAuth2.

In all, authentication implementations can quickly leave you in a confusion of JWT, SCRAM, Access Tokens, Refresh Tokens, and Timeouts. So...

Ask Fantom Factory to write the authentication for you!

We can create a mini SkySpark extension for HTTP Client that adds a custom authentication mechanism just for you!

That way you get to keep your passwords safe in the SkySpark password store, hide them from debug logs, and know that it has all been programmed to the highest possible standards.

Contact Fantom Factory at ------------------------- for details.

HTTP Errors

Not every HTTP request returns as expected, sometimes the REST API server returns an error in the form of a HTTP Response Status Code.

A common code is 404 - Not Found but there are many more. A full list of status codes may be found on wikipedia.

When HTTP Client Ext encounters an error status code, generally a 4xx or 5xx code, then it throws an error. If you want to handle these error cases yourself, you can perform a standard try / catch and check the returned status code directly in the err.

try do
    afHttpGet(`https://fantom.org/error404`)

catch (err) do
    if (err->statusCode == 404)
        echo("Page not found!")
    else
        // perpetuate all other errors
        throw err
end

An alternative is to clear the default HTTP response handler for that particular status, then manually check the returned code yourself:

afHttpClear({statusCode:404})
afHttpGet(`https://fantom.org/error404`)

res : afHttpResponse()
if (res->statusCode == 404)
    echo("Page not found!")

Note that multiple HTTP status code handlers may be cleared at once using an x or * wildcard. For example, to stop following ALL redirects:

// disable redirect following
afHttpClear({statusCode:"30x"})

Debugging

It can be very useful to see exactly what HTTP traffic is being sent and received. Calling afHttpDebug() will turn on debugging; but just for the current session (in case you forget to turn it off again!).

With debugging on, HTTP requests and responses will be written to the console, giving full visibility as what's happening behind the scenes.

<afHttpClient.demo> {ext} [debug]

  HTTP Request:
  GET / HTTP/1.1
  Host: www.google.com
  User-Agent: afHttpClient/2.1.5 (Fantom/1.0.77.3103 win32-x86_64)

<afHttpClient.demo> {ext} [debug]

  HTTP Response:
  HTTP/1.1 200 OK
  Date: Fri, 22 Nov 2019 10:44:12 GMT
  Expires: -1
  Cache-Control: private, max-age=0
  Content-Type: text/html; charset=ISO-8859-1
  P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
  Content-Encoding: gzip
  Server: gws
  Content-Length: 5418
  X-XSS-Protection: 0
  X-Frame-Options: SAMEORIGIN
  Set-Cookie: 1P_JAR=2019-11-22-10; path=/; domain=.google.com

  <!doctype html>
  <html lang="en-GB">
    <head>
      <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
      <title>Google</title>
      ...
      ...
      ...
  </html>

All passwords and credentials set via afHttpAuth() will be masked (starred) out, keeping them safe.

Because requests may hold other sensitive data, afHttpDebug() is restricted to super users only.

To keep debugging turned on for longer, you can set the log level for HTTP Client Ext in the standard SkySpark Debug -> Log view.

SSL Handshake Exceptions

Should any of the HTTP functions throw an javax.net.ssl.SSLHandshakeException, then that signifies a problem with the underlying Java platform and not the HTTP Axon functions.

If you see this error in particular:

sys::IOErr: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException:
  PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException:
    unable to find valid certification path to requested target

And if running SkySpark on an older Java version, such as Java 8, then there may be an easy fix; import the SSL certificate from the target endpoint, into Java. SkySpark has a handy tool for doing just this; see SkySpark Trusted Certificates for details.

Understand that SSL is not a type of encryption, it's an ever growing suite of standards and technology. Meaning web clients (or in our case, Java) has to continually grow to support it all.

So make sure Java supports the required server cipher by inspecting the server's SSL certificate in a browser, or by visiting SSL Labs. Then visit Oracle's Java Security Providers to ensure the cipher is supported.

Also see StackOverflow's Problems connecting to HTTPS/SSL with Java client for other troubleshooting ideas.

Note that strong encryption ciphers are often disabled in Java by default. To enable them, you may need to install Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files.

Licensing

HTTP Client Ext is a commercial product and requires a licence to be purchased to run in production.

Licences for HTTP Client Ext are tied to a SkySpark installation / host ID. This means to use HTTP Client Ext on multiple production servers will require multiple licences; one per server. You will need to enter your SkySpark licence ID when purchasing, see Where is my SkyArc Installation ID? for details.

Purchased licences are available from the My Licences page and should be downloaded to the /var/lic/ directory of your SkySpark installation.

Minor updates to HTTP Client Ext may be provided free of charge, but major updates will require a new licence to be purchased.

The End-User License Agreement ("EULA") is contained with the downloaded .pod file.

For support and comments, please email ----------------------.

Evaluation Mode

If HTTP Client Ext does not find a valid licence, it enters into an evaluation mode whereby a maximum of 20 HTTP calls may be made; after which HTTP Client Ext will throw a licence fault.

Evaluation mode is designed for developers to try out various features of HTTP Client Ext to assess its suitability for purpose.

Disclaimer

The software is provided "AS IS" and the author disclaims all warranties with regard to this software including all implied warranties of merchantability and fitness. In no event shall the author be liable for any special, direct, indirect, or consequential damages or any damages whatsoever resulting from loss of use, data or profits, whether in an action of contract, negligence or other tortious action, arising out of or in connection with the use or performance of this software.

Special Mentions

Special thanks go to Jaap Balvers and Pieter van der Mijle of BAM Energy Systems for sponsoring, supporting, and beta testing HTTP Ext 1.0.

Thanks go to Ian Habermann of AFM for sponsoring the DIGEST authentication scheme.

Thanks go to Stehpen Frank of NREL for sound boarding the new v2.0 authentication features.

More thanks go to BAM Energy Systems for sponsoring the header, path, and query authentication schemes.