In this article I will show you how to use https://github.com/testingisdocumenting/webtau WebTau tool to write and run end-to-end tests for a Web App that has REST API, GraphQL API, Web UI and CLI.I will show you how to use REPL mode to speed up test development and tighten feedback loop. And I will show you how to use end-to-end testing to help you with documentation writing.Article is long so here is quick access to different parts you can use later. For now keep reading from the start.WebTau stands for Web Tests Automation. An open source tool, API, and a framework designed to simplify testing on multiple levels.What do I mean by multiple levels? Apps I develop often have these layers:REST/GraphQL API Web UI Command Line Database When I write end-to-end tests I test on one layer, and validate the outcome using another. I write a test for a command line tool and validate that CLI updates a REST resource. I write a test for Web UI and use GraphQL API to set up the initial data.WebTau lets you manipulate and validate various layers using consistent API and analyze your test results using rich reporting that captures everything you do.

Article is quite large, and it may take you 15+ minutes to finish.I am new to writing articles and English is my second language. If you have suggestions, it would be awesome if you create a https://github.com/testingisdocumenting/blog/issues GitHub Issue with a grammar/content suggestion or even create a https://github.com/testingisdocumenting/blog/blob/master/blog-content/articles/ultimate-end-to-end-test.md PR.I am for https://mobile.twitter.com/search?q=https%3A%2F%2Ftestingisdocumenting.org%2Fblog%2Fentry%2Fultimate-end-to-end-test Discussion on twitter if you have ideas/opinions you would like to share.Tool that is being discussed here is also on https://github.com/testingisdocumenting/webtau GitHub. It is in active development and is being used for testing production systems. It covers use-cases that I and other tool contributors are facing daily and we are always looking for new challenges to tackle. Issues, PRs, and Stars are welcome!

We are going to test Game Store product. It has Web UI where you can see what games are available.It has CLI tool to help admins to manage the list of games. gs-admin list List of games g1 Slay The Spire card rpg 20.0 g2 Civilization 6 strategy 60.0 g3 Doom fps 60.0 g4 Last Of Us 2 adventure drama 60.0 g5 Inside adventure 10.0 g6 Hearthstone card 0.0 g7 Division 2 shooter rpg 30.0 g8 Assassin Creed Odyssey rpg 40.0 It has GraphQL and REST API to manage data. def query = ''' query { game(id: "g1") { title type } } ''' And it has a database to manage all the data.Below we are going to see how WebTau lets you seamlessly test on different layers and use other layers to help with data validation.

Before writing a test for Game Store, we are going to use https://jsonplaceholder.typicode.com/ JSON Placeholder website to demo basic WebTau parts.Our goal is to get and validate a response from this end point: https://jsonplaceholder.typicode.com/todos/1 To declare a test in WebTau we need to create a file and use scenario keyword to define test logic: scenario('basic http invocation') { http.get('https://jsonplaceholder.typicode.com/todos/1') { body.title.should == 'delectus aut autem' // automatic mapping of a field name to a JSON response completed.should == false // specifying body is optional } } Note: Most of the time we are going to use WebTau as both API and command line runner. You can use WebTau with JUnit5 and JUnit4 and pure Java if you prefer.To run a test, assuming webtau is in PATH use: webtau httpBasics.groovy scenario basic http invocation (httpBasics.groovy) > executing HTTP GET https://jsonplaceholder.typicode.com/todos/1 . body.title equals "delectus aut autem" body.title: actual: "delectus aut autem" <java.lang.String> expected: "delectus aut autem" <java.lang.String> (16ms) . body.completed equals false body.completed: actual: false <java.lang.Boolean> expected: false <java.lang.Boolean> (0ms) . header.statusCode equals 200 header.statusCode: actual: 200 <java.lang.Integer> expected: 200 <java.lang.Integer> (0ms) response (application/json; charset=utf-8): { "userId": 1, "id": 1, "title": __"delectus aut autem"__, "completed": __false__ } . executed HTTP GET https://jsonplaceholder.typicode.com/todos/1 (359ms) WebTau captures everything that happens in a test:commands there were ran assertions that were made (both passed and failed) values that were checked ( __ in the console output) WebTau generates rich HTML report with all the captured data at the end tests run. We are going to look at the report in details later, for now here is a screenshot of a generated report based on the tests below.

To avoid writing full url in our tests let's define a base url for our service. We are also going to define a base url UI. And we are going to define a separate dev environment: url = "http://localhost:8080" // base url for all http and browser requests browserUrl = "http://localhost:3000" // base url for browser open commands. Overrides url above if both are present browserId = "chrome" // specifying browser to use for browser based operations browserWidth = 900 browserHeight = 600 environments { dev { // optional overrides for the configs for dev environment url = "http://dev-server:8080" browserUrl = "http://dev-server:8080" } } package basicScenarios scenario('implicit config usage') { http.post("/relative-url") // implicit usage of core url config value browser.open("/test") // implicit usage of browser url and browser id } You can override config values using CLI params. Use --env=<value> to select an environment, i.e. a different set of config values: webtau scenarios/* --env=dev --url=http://override-value --browserId=chrome

Let's test Game Store API to register and check a game by its id. http://localhost:8080/api/game scenario('register new game') { http.post("/api/game", [ id: "g1", // define payload as a map title: "Slay The Spire", type: "card rpg", priceUsd: 20]) http.get("/api/game/g1") { // request newly created game as GET id.should == "g1" title.should == "Slay The Spire" // direct access to response JSON title property type.should == "card rpg" priceUsd.should == 20 } } Did you notice that request to POST and response from GET looks the same? Let's extract payload into a variable and re-use it for both request payload and response validation. This time we are not going to pass id to the request and instead will rely on the server to generate a new ID. package scenarios.gamestore // optional package declaration for IDE happiness import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* // optional single import for IDE autocomplete scenario('register new game') { def payload = [title: "Slay The Spire", // no ID specified this time type: "card rpg", priceUsd: 20] def id = http.post("/api/game", payload) { return id // we grab id from response, return keyword is optional } http.get("/api/game/${id}") { // using id for subsequent request body.should == payload // re-using POST payload to assert GET response. Only values in payload will be validated } }

End-point /api/game was not requiring authentication. It was done for example’s sake to make it simpler to approach. Now let's test an end-point that requires auth. scenario('save preferences without auth') { http.put('/api/user-preferences', [favoriteGenre: 'RPG']) { statusCode.should == 403 // forbidden, as this end-point requires authentication } } To authenticate a user our system relies on Bearer token. Let's generate a token and explicitly pass it via header parameter. scenario('save preferences with explicit auth') { def token = generateToken('user-a') // generates token using our system underlying auth system http.put('/api/user-preferences', http.header([Authorization: "Bearer ${token}"]), // explicitly pass Bearer token [favoriteGenre: 'RPG']) { userId.should == 'user-a' // make sure server recognized token and properly authenticated user } }

WebTau provides a way to implement implicit authentication for your requests. Before we get there, we need to cover Persona concept. Persona sets the context for config values and code execution. userId = 'NA' // custom config value defined in the default context personas { // persona config section, similar to environments config section Alice { userId = "uid-alice" // Alice's specific config value } Bob { userId = "uid-bob" // Bob's specific config value } } def Alice = persona('Alice') // define Alice persona def Bob = persona('Bob') // define Bob persona scenario('persona demo') { println cfg.userId // custom value from config Alice { println cfg.userId // custom value from config in Alice's context } Bob { println cfg.userId // custom value from config in Bob's context } } scenario persona demo (personaDemo.groovy) NA uid-alice uid-bob

Instead of explicitly passing header to each http call we will execute calls in the context of a persona. scenario('save preferences with personas auth') { Alice { // Alice's context http.put('/api/user-preferences', [favoriteGenre: 'RPG']) { userId.should == 'uid-alice' // validating that we updated the right user favoriteGenre.should == 'RPG' } } Bob { // Bob's context http.put('/api/user-preferences', [favoriteGenre: 'Strategy']) { userId.should == 'uid-bob' // validating that we updated the right user favoriteGenre.should == 'Strategy' } } } Console output (and report that we are going to look at later) captures what steps were executed in what context. running: scenarios/gamestore/userPreferencesRest.groovy save preferences with personas auth scenario save preferences with personas auth (userPreferencesRest.groovy) > Alice executing HTTP PUT http://localhost:8080/api/user-preferences request (application/json): { "favoriteGenre": "RPG" } > Alice mapping operation id . Alice mapped operation id as "PUT /api/user-preferences" (0ms) . Alice body.userId equals "uid-alice" body.userId: actual: "uid-alice" <java.lang.String> expected: "uid-alice" <java.lang.String> (1ms) . Alice body.favoriteGenre equals "RPG" body.favoriteGenre: actual: "RPG" <java.lang.String> expected: "RPG" <java.lang.String> (0ms) . Alice header.statusCode equals 200 header.statusCode: actual: 200 <java.lang.Integer> expected: 200 <java.lang.Integer> (0ms) > Alice validating request and response . Alice validated request and response (5ms) response (application/json): { "userId": __"uid-alice"__, "favoriteGenre": __"RPG"__ } . Alice executed HTTP PUT http://localhost:8080/api/user-preferences (164ms) > Bob executing HTTP PUT http://localhost:8080/api/user-preferences To make sure our PUT worked as intended we are going to GET user preferences in different Persona contexts. scenario('read preferences with personas auth') { Alice { // Alice's context http.get('/api/user-preferences') { favoriteGenre.should == 'RPG' // we get back the Alice's favorite genre } } Bob { // Bob's context http.get('/api/user-preferences') { favoriteGenre.should == 'Strategy' // we get back the Bob's favorite genre } } } How does it work behind the covers? WebTau allows you to define an implicit HTTP Header Provider that can inject header values into each request. httpHeaderProvider = HttpHeaderProvider.&provide // implicit header provider userId = '' // explicitly set default userId to be an empty string personas { Alice { userId = 'uid-alice' // Alice's system specific user id } Bob { userId = 'uid-bob' // Bob's system specific user id } } Our custom implementation of HTTP Header Provider checks cfg.userId value and when it is present, it will be used to generate Bearer token and inject into ongoing request.Note: cfg.userId is only set in the context of Bob or Alice . Outside persona context userId is an empty value. package auth import org.testingisdocumenting.webtau.http.HttpHeader import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* class HttpHeaderProvider { static provide(String fullUrl, String url, // provide method will be called for every HTTP request HttpHeader httpHeaders) { def userId = cfg.userId // grab custom userId config value, depends on the Persona context. Based on our current config, it is empty outside persona context. return userId ? httpHeaders.with("Authorization", // create new http header by taking a header passed by a test explicitly and adding Authorization key-value "Bearer ${generateToken(userId)}"): httpHeaders // original header passed by a test } // generate token based on your auth system, dummy impl for the demo purposes only private static def generateToken(String userId) { return userId.bytes.encodeBase64().toString() } }

We covered how to use http. namespace to test REST based API layer. Let's move to graphql. namespace to test GraphQL based API.Game Store provided a few queries and mutations: type Query { games: [Game] game(id: String): Game userPreferences: UserPreferences } type Mutation { createGame(id: String!, title: String!, type: String!, priceUsd: Int!): Game updateUserPreferences(favoriteGenre: String): UserPreferences } type Game { id: String title: String type: String priceUsd: Int } type UserPreferences { userId: String favoriteGenre: String } To start, we define a query to fetch a game by hardcoding id in the query. def query = ''' query { game(id: "g1") { title type } } ''' scenario('query game') { graphql.execute(query) { game.title.should == "Slay The Spire" // explicit access through query name title.should == "Slay The Spire" // implicit access in case of the single query } } GraphQL allows to define a query with parameters: def queryWithParams = ''' query game($id: String!) { game(id: $id) { title type } } ''' WebTau provides a way to pass parameter values as a Map : scenario('query game with param') { graphql.execute(queryWithParams, [id: 'g1']) { title.should == "Slay The Spire" } }

As in REST based user preference example above, updateUserPreferences mutation requires authentication. def mutation = ''' mutation updateUserPreferences($favoriteGenre: String!) { updateUserPreferences(favoriteGenre: $favoriteGenre) { userId favoriteGenre } } ''' Unlike REST based API, the auth error appears in errors field on a response and not via statusCode : scenario('save preferences without auth') { graphql.execute(mutation, [favoriteGenre: 'CRPG']) { errors.message.should == ['forbidden'] // shortcut to take a message from each error in the list } } To do auth explicitly, similar to REST API, we can pass header as a parameter to graphql.execute : scenario('save preferences with explicit auth') { def token = generateToken('user-a') graphql.execute(mutation, [favoriteGenre: 'CRPG'], http.header([Authorization: "Bearer ${token}"])) { userId.should == 'user-a' } }

Persona based authentication for GraphQL works exactly like REST API based one. scenario('save preferences with personas auth') { Alice { // Personas we defined and used for HTTP REST API graphql.execute(mutation, [favoriteGenre: 'RPG']) { userId.should == 'uid-alice' // make sure system picked Alice as a user favoriteGenre.should == 'RPG' } } Bob { graphql.execute(mutation, [favoriteGenre: 'Strategy']) { userId.should == 'uid-bob' // make sure system picked Bob as a user favoriteGenre.should == 'Strategy' } } } httpHeaderProvider = HttpHeaderProvider.&provide // implicit header provider userId = '' // explicitly set default userId to be an empty string personas { Alice { userId = 'uid-alice' // Alice's system specific user id } Bob { userId = 'uid-bob' // Bob's system specific user id } } package auth import org.testingisdocumenting.webtau.http.HttpHeader import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* class HttpHeaderProvider { static provide(String fullUrl, String url, // provide method will be called for every HTTP request HttpHeader httpHeaders) { def userId = cfg.userId // grab custom userId config value, depends on the Persona context. Based on our current config, it is empty outside persona context. return userId ? httpHeaders.with("Authorization", // create new http header by taking a header passed by a test explicitly and adding Authorization key-value "Bearer ${generateToken(userId)}"): httpHeaders // original header passed by a test } // generate token based on your auth system, dummy impl for the demo purposes only private static def generateToken(String userId) { return userId.bytes.encodeBase64().toString() } } Let's make system reflects the changes performed with mutations def query = ''' query { userPreferences { userId favoriteGenre } } ''' scenario('read preferences with personas auth') { Alice { graphql.execute(query) { favoriteGenre.should == 'RPG' // making sure correct data is returned back } } Bob { graphql.execute(query) { favoriteGenre.should == 'Strategy' // Strategy games are not bad } } }

We started with defining personas in-place within each test file like this: import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* def Alice = persona('Alice') // define Alice persona def Bob = persona('Bob') // define Bob persona scenario('persona demo') { println cfg.userId // custom value from config Alice { println cfg.userId // custom value from config in Alice's context } Bob { println cfg.userId // custom value from config in Bob's context } } As we start using Personas for multiple tests and multiple files it makes sense to define Personas once and reference them in tests as needed. package personas // user defined package name/directory import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* class Personas { // class name can be anything public static final def Admin = persona('Admin') public static final def Alice = persona('Alice') public static final def Bob = persona('Bob') } To use defined personas we leverage Java/Groovy static import: package basicScenarios import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* import static personas.Personas.* // import all available personas to be in scope scenario('persona demo') { println cfg.userId Alice { // Alice is taken from personas.Personas.* println cfg.userId } Bob { println cfg.userId } }

We looked at how to test REST and GraphQL based APIs. Now let's move on to testing Web UI. Before we jump into testing Game Store UI, we will do a classic test of Google Page.Our test will enter "test" value into the search box and, wait for the results to show up and pick a result based on a regular expression: scenario('basic browser interaction') { browser.open("https://google.com") $('input[title="Search"]').setValue("test\n") // set value based on the underlying component $('h3').count.waitTo >= 1 // explicit, simplified wait on condition $("h3").get(~/Speedtest.*Ookla/).click() // example of filter based on regular expression browser.url.full.should == ~/speedtest.net/ // full url assertion } As with REST and GraphQL, WebTau console output captures all the actions that happen, and how much time each action took: scenario basic browser interaction (browserBasics.groovy) > initializing webdriver for chrome . initialized webdriver for chrome (2s 39ms) > opening https://google.com . opened https://google.com (593ms) > setting value test\n to by css input[title="Search"] > clearing by css input[title="Search"] . cleared by css input[title="Search"] (19ms) > sending keys test\n to by css input[title="Search"] . sent keys test\n to by css input[title="Search"] (906ms) . set value test\n to by css input[title="Search"] (945ms) > documentation capturing screenshot as browser-basic . documentation captured screenshot as /Users/mykolagolubyev/work/testingisdocumenting/blog/game-store/game-store-tests/game-store-tests-e2e/src/test/groovy/basicScenarios/../../../../target/doc-artifacts/browser-basic.png (440ms) > waiting to count of by css h3 to be greater than or equal to 1 . count of by css h3 greater than or equal 1 count: actual: 13 <java.lang.Integer> expected: greater than or equal to 1 <java.lang.Integer> (18ms) > clicking by css h3, element(s) with regexp Speedtest.*Ookla . clicked by css h3, element(s) with regexp Speedtest.*Ookla (646ms) > expecting full page url of browser to equal pattern /speedtest.net/ . full page url of browser equals pattern /speedtest.net/ full page url: actual string: https://www.speedtest.net/ expected pattern: speedtest.net (4ms) Note: Both passed and failed assertions are captured.

The first Game Store page we are going to test is a landing page. It shows available games and let you filter based on text or price.We covered how to use http. layer to create new games. We can use that to prepare a setup for our UI test: package scenarios.gamestore import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* scenario('landing page') { http.post("/api/game", [id: "g1", title: "Slay The Spire", type: "card rpg", priceUsd: 20]) // pre-create test data using HTTP http.post("/api/game", [id: "g2", title: "Civilization 6", type: "strategy", priceUsd: 60]) http.post("/api/game", [id: "g3", title: "Doom", type: "fps", priceUsd: 40]) browser.open("/") // open landing page, relying on url and/or browserUrl def titles = $('[class*="GamesList_title"]') // define titles as page element selected by css titles.waitTo == ['Civilization 6', 'Doom', 'Slay The Spire'] // wait for games titles to show up and match our expectations http.delete("/api/game/g3") // delete a single game using HTTP titles.waitTo == ['Civilization 6', 'Slay The Spire'] // wait for UI to reflect changes } Let's test the filters: scenario('filter by text') { browser.reopen("/") $("#filter").setValue("civ") // use setValue abstraction to set value inside input box $('[class*="GamesList_title"]').waitTo == ['Civilization 6'] // wait for changes to be reflected } scenario('filter by price') { browser.reopen("/") $("#below60").setValue(true) // use setValue abstraction to set value for a checkbox $('[class*="GamesList_title"]').waitTo == ['Doom', 'Slay The Spire'] // wait for changes to be reflected } Note: setValue is used for both text input box and check box. WebTau will figure out what actions to perform based on the underlying form element.

If you have experience in writing UI tests, you may have heard about https://martinfowler.com/bliki/PageObject.html Page Object pattern. Basic idea is to separate what user can see or do from technical details of how to simulate the actions or query the values.If we take a look at the code in our UI test we will see that details of how to locate elements on the UI are exposed. Any time we change id element or class names, we risk breaking our tests.Warning: Tests should not be broken if only implementation details has changed. $("#filter").setValue("civ") // exposing css selector to the test is going to haunt you later $('[class*="GamesList_title"]').waitTo == ['Civilization 6'] // class based selection is more likely to become out of sync To implement page object in WebTau we will use a regular Java/Groovy/Kotlin class: package pages import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* class LandingPage { def titles = $('[class*="GamesList_title"]') def filterText = $("#filter") // lazy declaration for future usage, no actual attempt to find element is performed def filterBelow60 = $("#below60") def labelBelow60 = $("span").get('Below $60') // filters chaining is also lazy def adminMessage = $("#admin-message") def reopen() { browser.reopen("/") } } Since our page object class is stateless and elements declarations are lazy, we can precreate instances of all the page objects in one place: package pages class Pages { static final def maintenance = new MaintenancePage() // pre-creating stateless instances static final def login = new LoginPage() static final def landing = new LandingPage() // for all the pages our test suite needs to have access static final def userPreferences = new UserPreferencesPage() } package scenarios.gamestore import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* import static pages.Pages.* // import all the pages into test scope scenario('filter by text') { landing.reopen() // set value using input defined in the page object landing.filterText.setValue("civ") landing.titles.waitTo == ['Civilization 6'] } scenario('filter by price') { landing.reopen() // set checkbox value using input defined in the page object landing.filterBelow60.setValue(true) landing.titles.waitTo == ['Doom', 'Slay The Spire'] }

You saw waitTo in a couple of places already. Let's pause on the topic.Question: Why do we need waitTo and can we instead make should do the waiting?We could. However, semantics are important. I think it is important to distinct between an immediate result, and a result that becomes available over time.Are our users going to see the result right away? Is database going to have the result right away? Is command line going to print that line right away?Additionally waitTo is not specific for browser. layer. It can be used to wait on files content, console output, database data, etc. // waiting on query to finally return a value (eventual consistency) db.query("select * from games").waitTo contain([title: "Slay The Spire"]) // waiting of file content to contain a line fs.textContent("my-file.txt").waitTo contain('important line') def command = cli.runInBackground("admin-tool") Pages.login.login("myname", "mypassword") // waiting on command line tool to print important message command.output.waitTo contain("<myname> logged in") Testing Is DocumentingTesting is Documenting is my moto. WaitTo semantics allow you to look at the test and get a better understanding of how system behaves. This is an example of a more subtle, more common definition of how tests can help you to understand the system.Later I will show you how to make tests help you with the actual documentation in a more explicit manner.

Game Store have a page where users can update their user preferences.If we go to that page directly, we will be redirected to the login page instead.In order to test user preferences page, we have to choices:handle login explicitly handle login implicitly scenario('user preferences redirects to login') { userPreferences.open() // open user preferences page login.name.waitTo beVisible browser.url.ref.should == '/login' // but landed on login page login.login('uid-test', 'dummy-password') // explicitly enter user name and password userPreferences.userId.waitTo == 'uid-test' // after redirect we can see user preferences userPreferences.favoriteGenre.should == '' } package pages import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* class UserPreferencesPage { def userId = $('#user-id') // user id to assert on def favoriteGenre = $('#favorite-genre') // genre to validate/set def saveButton = $('#save') def saveResultMessage = $('#save-result') def open() { browser.reopen("/#/user") } def save() { // exposed action - regular method saveButton.click() } } package pages import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* class LoginPage { def name = $('#name') // name to wait on def password = $('#password') def loginButton = $('#login') void login(name, password) { // action to login with this.name.setValue(name) this.password.setValue(password) loginButton.click() } } Explicit login approach works well if we go the page once, and we know that we haven't performed login yet. If on the other hand we need to get to that page as part of other scenario, we may face the case where we have already logged-in, and the code that assumed logic redirection will fail.To fix that we will need to add additional if-else logic that may complicate things a bit.Alternatively we can use an implicit login with Personas.

Browser Implicit Auth in also Persona based. scenario('implicit login with persona') { Alice { // In the Alice's context http.put('/api/user-preferences', [favoriteGenre: 'RPG']) // prepare data to assert on - another example of http/browser test combination userPreferences.open() // open user preferences userPreferences.userId.waitTo == 'uid-alice' // we expect user preferences screen, not the login screen userPreferences.favoriteGenre.should == 'RPG' // our assertion matches what we set using REST API } } Let's now update preferences through UI and validate them using REST API: scenario('change preferences through UI') { Bob { http.put('/api/user-preferences', [favoriteGenre: 'Sim']) } // Set Bob's preference via HTTP Alice { userPreferences.open() userPreferences.favoriteGenre.waitTo beVisible() // wait for UI to load data from REST endpoint userPreferences.favoriteGenre.setValue('CRPG') userPreferences.save() // save user preferences userPreferences.saveResultMessage.waitTo == 'Saved' // wait for visual clue to appear } Bob { // In Bob's context http.get('/api/user-preferences') { favoriteGenre.should == 'Sim' // Bob's preferences were not affected by UI } } Alice { // In Alice's context http.get('/api/user-preferences') { favoriteGenre.should == 'CRPG' // Genre is the one we set } } } To make implicit auth work we need to implement browserPageNavigationHandler . A handler that will be called for each page open event. package auth import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* class BrowserOpenHandler { private static STORAGE_KEY = 'authToken' static void handleOpen(passedUrl, fullUrl, currentUrl) { // will be called for every page open def userId = cfg.userId // take user from config based on current persona if (!userId || browser.localStorage.getItem(STORAGE_KEY)) { // if no user or token is inside storage, we don't need to do anything return } browser.localStorage.setItem(STORAGE_KEY, generateToken(userId)) // set a new auth token generated based on current persona browser.reopen(fullUrl) // re-open original page } // generate token based on your auth system, dummy impl for the demo purposes only private static def generateToken(String userId) { return userId.bytes.encodeBase64().toString() } } Note: Game Store UI code uses local storage to manage auth token. In your app it could be session storage or cookies. browserPageNavigationHandler = BrowserOpenHandler.&handleOpen // implicit page open handler userId = '' // explicitly set default userId to be an empty string personas { Alice { userId = 'uid-alice' // Alice's system specific user id } Bob { userId = 'uid-bob' // Bob's system specific user id } } Question: How do we maintain different localStorage for different Personas?WebTau maintains a browser per persona . In the examples above we have a total of two browsers during test run:Default browser Alice's browser

Now that we know how to use Persona to manage multiple browsers, we can write a test for the last part of Game Store UI - admin page to send messages to visitors.Admin page allows to send a message to all visitors via WebSocket.In order to test this we will need two browsers: one to send a message and another to receive a websocket event.Let's start with defining a page object for the new page: package pages import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* class MaintenancePage { def message = $('#message') def sendMessage = $('button').get('send message') def reopen() { browser.reopen("/#/admin") } } package scenarios.gamestore import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* // single convenient import for IDE auto complete import static pages.Pages.* // convenient access to all page objects import static personas.Personas.* // convenient access to all Personas scenario('sending admin message') { landing.reopen() // open landing page in a default browser def message = 'Shop is going to be closed for maintenance' Admin { maintenance.reopen() // open maintenance page in Admin's browser maintenance.message.setValue(message) maintenance.sendMessage.click() // send notification message } landing.adminMessage.waitTo == message // in the default browser, wait for the message to arrive }

Another layer to explore is Database. Database layer can be used to prepare test data. It can also be used to validate data persistence after REST/GraphQL API calls.Warning: Database layer is often considered to be implementation details, and most of your tests should not use it directly. However, it can be useful to validate that REST POST method did modify data in the database and didn't persist data in cache only. scenario('http post and db list games') { db.update("delete from Game") // wipe out table http.post("/api/game", [id: "g1", title: "Slay The Spire", type: "card rpg", priceUsd: 20]) // create two games using HTTP http.post("/api/game", [id: "g2", title: "Civilization 6", type: "strategy", priceUsd: 60]) def GAME = db.table("GAME") GAME.should == ["*ID" | "TITLE"] { // make sure DB reflects changes __________________________ "g1" | "Slay The Spire" "g2" | "Civilization 6"} } scenario('db write and http list games') { db.update("delete from Game") def GAME = db.table("GAME") // populate GAME table with two rows GAME << ["ID" | "TITLE" | "TYPE" | "PRICE_USD"] { ____________________________________________________ "g1" | "Slay The Spire" | "card rpg" | 20 "g2" | "Civilization 6" | "strategy" | 60 } http.get("/api/game") { // expect body to contain a list of two games body.should == [ "*id" | "title" ] { ___________________________ "g2" | "Civilization 6" "g1" | "Slay The Spire" } } }

Command line interface is our final layer to test. Before we test admin app, let's see how cli. works on a simple command: cli.run("ls -l") cli.run("ls -l") { output.should contain("webtau.cfg.groovy") } total 1032 -rw-r--r-- 1 mykolagolubyev staff 586 Jun 13 17:58 browserBasics.groovy -rw-r--r-- 1 mykolagolubyev staff 302 Jun 13 17:57 cliBasics.groovy -rw-r--r-- 1 mykolagolubyev staff 210 Jun 13 17:57 configAccess.groovy -rw-r--r-- 1 mykolagolubyev staff 402 Jun 13 17:57 httpBasics.groovy -rw-r--r-- 1 mykolagolubyev staff 487 Jun 13 17:57 personaDemo.groovy -rw-r--r-- 1 mykolagolubyev staff 355 Jun 13 17:57 personaReUseDemo.groovy -rw-r--r-- 1 mykolagolubyev staff 1008 Jun 13 17:57 tableDataDemo.groovy -rw-r--r-- 1 mykolagolubyev staff 658 Jun 13 17:57 waitToDemo.groovy -rw-r--r-- 1 mykolagolubyev staff 567 Jun 13 17:58 webtau.cfg.groovy -rw-r--r-- 1 mykolagolubyev staff 327 Jun 13 17:57 webtau.persona.cfg.groovy -rw-r--r-- 1 mykolagolubyev staff 486878 Oct 4 12:27 webtau.report.html Our admin CLI tool works similar ls and list games available in the system: List of games 43daf57f-ca2d-477b-8853-dc7eeec30c7c Slay The Spire card rpg 20.0 0e731811-d2a5-4a4a-9431-e16e0e9f3ed4 Civilization 6 strategy 60.0 We still develop the tool and it is not available in PATH yet, so we will need to use a relative path to run it. Exposing the path to tests will make our tests brittle in the similar way how exposing UI elements definition does: any time we change the tool location we will have to change tests using the tool.Similar to UI Page Object idea, we declare CLI tools our tests need access to in a separate file/class: package clicommands import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* class CliCommands { static final def adminTool = cli.command( // lazy declaration of our CLI command "java -jar ${cfg.workingDir}/../../../../../game-store-cli/target/admin-tool-jar-with-dependencies.jar") // command line location is subject to change and should not be exposed to a test } package scenarios.gamestore import static clicommands.CliCommands.* // convenient access to all declared command line tools import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* scenario('list games using CLI') { // cleaning up DB db.update("delete from GAME") // run our command line adminTool.run { output.should contain("List of games") // expect a title to be present, and nothing else output.numberOfLines.should == 1 } http.post("/api/game", [title: "Slay The Spire", type: "card rpg", priceUsd: 20]) // pre-create data we need for the test using http layer http.post("/api/game", [title: "Civilization 6", type: "strategy", priceUsd: 60]) adminTool.run { output.should contain("Slay The Spire") // make sure output contains games we created output.should contain("Civilization 6") } }

We have seen TableData usage when we worked with Database and when we validated complex REST API output. Let's take a closer look at it. TableData is a core WebTau data type. Think of it as a list of maps with extra functionality. You can assign TableData to a variable, pass it around, return from functions and use for assertions. Data is available at runtime, and it is not a compile-time construct.Here is an example of usage of extra TableData features to generate test data: scenario('table data demo') { def increment = cell.above + 1 // create increment cell function to auto increase value def tableData = ["id" | "name" | "score" | "type" | "disabled"] { ____________________________________________________________________________ cell.guid | "A" | 100 | "T0R" | false // cell.guid generates guid cell.guid | "B" | cell.above | "MBR" | true // cell.above reuses value from the row above cell.guid | "C" | increment | permute("PO", "AC") | permute(true, false) } // permute generates multiple rows based on the values passed println("table data:") println(tableData) println(tableData.collect { it.score }) // standard collection operations are available on table data } scenario table data demo (tableDataDemo.groovy) table data: :id |name|score|type |disabled: .______________________________________.____._____._____.________. |"10a7feee-b965-4670-bc1f-b832ca012f61"|"A" |100 |"T0R"|false | .______________________________________.____._____._____.________| |"06a9e6d6-746a-4f47-b983-e09d9dd44517"|"B" |100 |"MBR"|true | .______________________________________.____._____._____.________| |"50138e3e-a815-4a50-83a6-7db1c63fa2eb"|"C" |101 |"PO" |true | .______________________________________.____._____._____.________| |"e5f3989f-1d71-4698-9646-15ff54ed9d18"|"C" |102 |"AC" |true | .______________________________________.____._____._____.________| |"216e0b81-9b70-4259-b205-f8642a995de0"|"C" |103 |"PO" |false | .______________________________________.____._____._____.________| |"1e97ee07-0848-4950-a1fb-c78bc9b843a8"|"C" |104 |"AC" |false | .______________________________________.____._____._____.________| [100, 100, 101, 102, 103, 104]

It is worth noting that TableData is not specific to Groovy command line runner. In fact it can be used for example as JUnit5 test factory: package com.example.junit5 import org.junit.jupiter.api.TestFactory class DynamicTestsGroovyTest { @TestFactory // Junit5 test factory that expects Stream<DynamicTest> def "test factory example"() { ["price" | "quantity" | "outcome"] { ___________________________________ 10 | 30 | 300 -10 | 30 | -300 }.test { // test is a what generates DynamicTest from each row PriceCalculator.calculate(price, quantity).should == outcome } } } Moreover, TableData is not specific to Groovy and you can use Java syntax to define an instance: package com.example.junit5; import org.junit.jupiter.api.DynamicTest; import org.junit.jupiter.api.TestFactory; import org.testingisdocumenting.webtau.data.table.TableData; import org.testingisdocumenting.webtau.junit5.DynamicTests; import java.util.stream.Stream; import static org.testingisdocumenting.webtau.WebTauCore.*; // convenient single import for unit tests class DynamicTestsJavaTest { TableData useCases = table("price", "quantity", "outcome", // java table definition ______________________________, 10 , 30, 300, -10 , 30, -300); @TestFactory public Stream<DynamicTest> testFactoryExample() { return DynamicTests.fromTable(useCases, r -> { // generate DynamicTest per row int price = r.get("price"); int quantity = r.get("quantity"); int outcome = r.get("outcome"); actual(PriceCalculator.calculate(price, quantity)).should(equal(outcome)); }); } }

Since we are talking about JUnit and Java, let me show you an example of REST API test using JUnit5 and Java syntax: package com.example.junit5; import org.junit.jupiter.api.Test; import org.testingisdocumenting.webtau.junit5.WebTau; import java.util.Map; import static org.testingisdocumenting.webtau.WebTauDsl.*; // convenient import for Java @WebTau // optional annotation to include this test into web report public class PostGetJavaTest { @Test public void registerNewGame() { Map<String, Object> payload = aMapOf( // define payload as a map "id", "g1", "title", "Slay The Spire", "type", "card rpg", "priceUsd", 20); http.post("/api/game", payload); // same post as in Groovy http.get("/api/game/g1", (header, body) -> { body.get("title").should(equal("Slay The Spire")); // validating title field in the response }); } } In case of JUnit runner, WebTau reads config values from webtau.properties resource file: url=http://localhost:8080

As I mentioned at the beginning, WebTau captures a lot of information. Information is printed to the console, but also stored in a rich HTML report.Generated report is a self-contained HTML file that can be emailed, slacked or copied to a network drive. You don't need to have a server to look at the report. Open it in a browser, and you will get an interactive mini app.Report Summary page consist ofTests Runtime - shows overall time spend Tests success ratio - how many failed vs passed HTTP Coverage - how many REST API operations are covered (WebTau uses optional Open API spec url to discover all the operations) Tests run Ratio - how many skipped vs ran HTTP calls time - high level stats on HTTP based APIs Let's click on ( 6 ) to get more details about skipped operations:Report switched from Tests view to HTTP Calls view ( 1 ). You can see the list of operations we yet to test on the left ( 2 ). Number of items on the left matches the number in the summary card ( 3 ). We can use panel at the bottom ( 4 ) to switch between skipped, total, passed operations or tests.Let's navigate from Summary tab to Overal HTTP Performance tab. The screen shows high level picture of how performant our server is under test load. While it is not a true performance test, it may give you an initial glance into things that may need a closer look.HTTP Operations Performance tab provides a performance information based on Open API defined operations.Let's get back to tests view ( 1 ) and select a test ( 2 ). First screen we are going to see is a test Summary ( 3 ) with high level information on time taken.If a test performed any HTTP calls, you're going to see the HTTP Calls tab ( 1 ). It contains every HTTP call performed with captured request, response and assertions ( 2 and 3 ) information.If a test performed any CLI calls, you're going to see the CLI Calls tab ( 1 ). It contains every CLI call performed with captured command, output and assertions ( 2 ) information.Every test also has Steps tab ( 1 ) that contains every step test performed, time it took, and what Persona if any performed the step ( 2 and 3 )

REPL stands for read-eval-print-loop. It is an interactive computer programming environment that helps with prototyping.You may have already used REPL if you usedJupyter Notebook iPython R MatLab webtau:000> 2 + 2 ===> 4 webtau:000> a = 5 ===> 5 webtau:000> a + 3 ===> 8 Question: Why are we talking about REPL in the context of end to end testing?We want to have a fast feedback loop.Browser, Servers, DB setup takes time Preserving context End to end tests - slow feedback REPL - speedup feedback loop The faster feedback the happier you are

Writing UI tests can be time-consuming. One of the reason is it takes time to open a browser, run your test, add an extra line to the test, repeat.Instead we can run webtau in repl mode like this webtau repl after that we can execute one command at a time, preserving context . created db datasource primary (62ms) > running DB query select * from Game on primary-db . ran DB query select * from Game on primary-db (17ms) ID , PRICE_USD, TITLE , TYPE "g1", 20.00, "Slay The Spire", "card rpg" "g2", 40.00, "Doom" , "fps" webtau:000> browser.open("https://google.com") > initializing webdriver for chrome . initialized webdriver for chrome (1s 101ms) > opening https://google.com . opened https://google.com (3s 74ms) webtau:000> $('input[title="Search"]') element is found: by css input[title="Search"] getText(): getUnderlyingValue(): webtau:000> $('input[title="Search"]').setValue("test\n") > setting value test\n to by css input[title="Search"] > clearing by css input[title="Search"] . cleared by css input[title="Search"] (16ms) > sending keys test\n to by css input[title="Search"] . sent keys test\n to by css input[title="Search"] (1s 287ms) . set value test\n to by css input[title="Search"] (1s 334ms) webtau:000> $('input[title="Search"]') element is not present: by css input[title="Search"] webtau:000> $('h3') element is found: by css h3 getText(): Speedtest by Ookla - The Global Broadband Speed Test getUnderlyingValue(): Speedtest by Ookla - The Global Broadband Speed Test count: 13 webtau:000> $('h3').count.waitTo >= 0 > waiting to count of by css h3 to be greater than or equal to 0 . count of by css h3 greater than or equal 0 count: actual: 13 <java.lang.Integer> expected: greater than or equal to 0 <java.lang.Integer> (4ms)

Database layer can be used to semi-automatically validate state of your system during experimentation. Or it can be used to quickly wipe or setup the data. I personally use it during active development to iterate faster. > mapping operation id . mapped operation id as "POST /api/game" (0ms) . header.statusCode equals 201 header.statusCode: actual: 201 <java.lang.Integer> expected: 201 <java.lang.Integer> (0ms) > validating request and response . validated request and response (1ms) response (application/json): { "id": "g2", "title": "Doom", "type": "fps", "priceUsd": 40 } . executed HTTP POST http://localhost:8080/api/game (123ms) webtau:000> db.query("select * from Game") > creating db datasource primary

One of WebTau superpowers is REPL mode. It allows you to combine test run and test experimentation. Run webtau in repl mode with existing test files you want to play with: webtau scenarios/gamestore/adminCliTool.groovy scenarios/gamestore/userPreferencesRest.groovy scenarios/gamestore/userPreferencesGraphQL.groovy scenarios/gamestore/userPreferencesUi.groovy repl webtau:000> ls Test files: 0 scenarios/gamestore/adminCliTool.groovy 1 scenarios/gamestore/userPreferencesRest.groovy 2 scenarios/gamestore/userPreferencesGraphQL.groovy 3 scenarios/gamestore/userPreferencesUi.groovy Once you list all test files, you can select one by https://testingisdocumenting.org/webtau/REPL/test-runs#test-file-selection either index or text webtau:000> s "userPreferencesRest" Test scenarios of scenarios/gamestore/userPreferencesRest.groovy: 0 save preferences without auth 1 save preferences with explicit auth 2 save preferences with personas auth 3 read preferences with personas auth After test file selection, you can run one or more scenarios on demand. webtau:000> r "save preferences with personas auth" > pinging http://localhost:8080/actuator/health > executing HTTP GET http://localhost:8080/actuator/health . header.statusCode equals 200 header.statusCode: actual: 200 <java.lang.Integer> expected: 200 <java.lang.Integer> (0ms) > validating request and response Path, http://localhost:8080/actuator/health is not found in OpenAPI spec . validated request and response (0ms) response (application/json): { "status": "UP" } . executed HTTP GET http://localhost:8080/actuator/health (4ms) . pinged http://localhost:8080/actuator/health (4ms) running: scenarios/gamestore/userPreferencesRest.groovy save preferences with personas auth scenario save preferences with personas auth (userPreferencesRest.groovy) > Alice executing HTTP PUT http://localhost:8080/api/user-preferences request (application/json): { "favoriteGenre": "RPG" } > Alice mapping operation id . Alice mapped operation id as "PUT /api/user-preferences" (0ms) . Alice body.userId equals "uid-alice" body.userId: actual: "uid-alice" <java.lang.String> expected: "uid-alice" <java.lang.String> (1ms) . Alice body.favoriteGenre equals "RPG" body.favoriteGenre: actual: "RPG" <java.lang.String> expected: "RPG" <java.lang.String> (0ms) . Alice header.statusCode equals 200 header.statusCode: actual: 200 <java.lang.Integer> expected: 200 <java.lang.Integer> (0ms) > Alice validating request and response . Alice validated request and response (5ms) response (application/json): { "userId": __"uid-alice"__, "favoriteGenre": __"RPG"__ } . Alice executed HTTP PUT http://localhost:8080/api/user-preferences (164ms) > Bob executing HTTP PUT http://localhost:8080/api/user-preferences When I write tests, I keep re-running ( r ) a current test under development, then experiment with a few lines in REPL mode, do some checks, update the test file and re-run it ( r ). WebTau will reload test file and pickup your changes.After each test run, context is preserved, browser is open in the last location, DB is up to date, and REST/GraphQL APIs are primed.Here is an example of running a db. query after a test run: webtau:000> db.query("select * from user_preferences") > running DB query select * from user_preferences on primary-db . ran DB query select * from user_preferences on primary-db (1ms) USER_ID , FAVORITE_GENRE "user-a" , "RPG" "uid-alice", "RPG" "uid-bob" , "Strategy"

When we document things, we try them out and make sure they work as intended. When we test things, we follow happy paths and edge cases.Happy path tests often cover what our users will do. Happy path tests also often match the things we document.Documentation is hard: Mostly manual labor Often becomes outdated Question: How do we make documentation easier to write and maintain?Artifacts captureWe already have things in our codebase that we can use to help with our product documentation. Example code snippets, GraphQL schema files, basic config files.By writing happy path tests we can add a few more to the list:CLI Outputs HTTP responses Screenshots The major part of this content was generated by running testsWebTau console outputs your saw, the REST/GraphQL API response, Game Store and Webtau Report screenshots, all of it was automatically generated by running tests.

When I was showing your snippets of code before I was hidding some code from your. Let's take a look. scenario('db write and http list games') { db.update("delete from Game") def GAME = db.table("GAME") GAME << ["ID" | "TITLE" | "TYPE" | "PRICE_USD"] { ____________________________________________________ "g1" | "Slay The Spire" | "card rpg" | 20 "g2" | "Civilization 6" | "strategy" | 60 } http.get("/api/game") { body.should == [ "*id" | "title" ] { ___________________________ "g2" | "Civilization 6" "g1" | "Slay The Spire" } } http.doc.capture("game-store-list-after-db") } http.doc.capture generates a directory with captured response, request, url, assertions, etc /api/game [ { "id" : "g1", "title" : "Slay The Spire", "type" : "card rpg", "priceUsd" : 20.0 }, { "id" : "g2", "title" : "Civilization 6", "type" : "strategy", "priceUsd" : 60.0 } ] ["root[0].id","root[0].title","root[1].id","root[1].title"] WebTau provides browser.doc.capture to capture screenshots for documentation purposes. scenario('filter by price') { landing.reopen() landing.filterBelow60.setValue(true) landing.titles.waitTo == ['Doom', 'Slay The Spire'] browser.doc.withAnnotations( browser.doc.badge(landing.filterText).toTheRight(), browser.doc.badge(landing.labelBelow60).toTheRight()) .capture('admin-ui-filter') } In addition to the screenshot .png file WebTau also captures annotations placement and annotations types: { "shapes" : [ { "type" : "badge", "text" : "1", "x" : 224, "y" : 88, "align" : "ToTheRight" }, { "type" : "badge", "text" : "2", "x" : 344, "y" : 87, "align" : "ToTheRight" } ], "pixelRatio" : 2 } In case of CLI cli.doc.capture captures the command that was run, its output and assertions performed package scenarios.gamestore import static org.testingisdocumenting.webtau.WebTauGroovyDsl.* scenario('list games using CLI') { db.update("delete from GAME") adminTool.run { output.should contain("List of games") output.numberOfLines.should == 1 } http.post("/api/game", [title: "Civilization 6", type: "strategy", priceUsd: 60]) adminTool.run { output.should contain("Civilization 6") } cli.doc.capture("list-games-cli") } java -jar /Users/mykolagolubyev/work/testingisdocumenting/blog/game-store/game-store-tests/game-store-tests-e2e/src/test/groovy/../../../../../game-store-cli/target/admin-tool-jar-with-dependencies.jar List of games 43daf57f-ca2d-477b-8853-dc7eeec30c7c Slay The Spire card rpg 20.0 0e731811-d2a5-4a4a-9431-e16e0e9f3ed4 Civilization 6 strategy 60.0

I used the captured information to generate the content of this blog/presentation. At my work I use captured artifacts to produce user guides. This approach makes guides to be always up-to date and validated.Here is an example of Game Store manual created with captured artifacts and https://testingisdocumenting.org/znai/ Znai documentation system.

We tested Game Store on multiple layers, using one layer to set-up and re-inforce tests on the other layers. We used consistent matchers and concepts like should , waitTo , Persona across layers. We saw how each step and assertions is captured by WebTau and written to console and rich HTML report. We saw how REPL can improve feedback loop and make you write tests faster. We saw how writing tests can generate artifacts to help you with writing User facing documentation.

WebTau - GitHub: https://github.com/testingisdocumenting/webtau https://github.com/testingisdocumenting/webtau WebTau - User Guide: https://testingisdocumenting.org/webtau https://testingisdocumenting.org/webtauZnai - Github: https://github.com/testingisdocumenting/znai https://github.com/testingisdocumenting/znai Znai - User Guide: https://testingisdocumenting.org/znai https://testingisdocumenting.org/znai