RFC 89: [testdriver] Add an execute_script function to testdriver.

[jgraham]

This provides a direct way to execute script in another browsing context.

[jgraham]

Gecko has something like this, but instead of passing in source text you pass in a function. Maybe we could have an API like

test_driver.execute_script(callable, {args: []})

e.g.

test_driver.execute_script(async (elem_id) => {
  await new Promise(resolve => setTimeout(resolve, 0));
  return document.getElementById(elem_id).innerHTML
}, {args: ["test"]})

If the function returns a promise, we always await the promise, if it returns a value we always just return the value.

On the backend, we'd toString() the function and json-encode the args and do something like

webdriver.execute_async_script(f"""
let callback = arguments[arguments.length - 1];
let result = ({function_string}).apply(null, {json.loads(args_array)});
Promise.resolve(result).then(callback);
""")

[Ms2ger]

async as an argument seems somewhat painful. Most likely, it'll always be a bare boolean constant at the call sites, which doesn't tend to be readable. Would adding a separate execute_async_script method be much more work?

[jgraham]

Agreed. I'm going to update the RFC with the proposal in #89 (comment)

[jgraham]

I updated this based on #89 (comment) But the more I think about it, the more worried I am that the fact that WebDriver is single-threaded and waiting on some complex script running in a remote context seems like it comes with a high risk of tests that have to be killed by the harness.

[gsnedders]

needs to be ```

[gsnedders]

This also implies that recursive calls will never work, right?

[jgraham]

Yes.

[hiroshige-g]

References in fn are evaluated on context, while args are evaluated on the caller context, right?
e.g.

execute_script(
    (arg) => [location.href, `${location.href}`, arg],
    [location.href, `${location.href}`],
    context)

would result in [the URL of context, the URL of context, the URL of the caller of execute_script, the URL of the caller of execute_script].

[jgraham]

Yes, that's right.

[fergald]

what is "body" here? Should this be "fn"?

[hiroshige-g]

I tested BFCache test migration to jgraham's draft PR.

It works mostly similarly to the JavaScript-based COEP dispatcher framework, and I made a wrapper class that keeps most of the tests as-is.
Diffs/wrappers needed are uploaded at https://chromium-review.googlesource.com/c/chromium/src/+/3082215.

Functionality/implementation needed in the TestDriver version:

• A mechanism to wait for target context to be created or to become active (i.e. navigation completion triggered by window.open, back-navigation and restoration from BFCache, etc.).
  I made WPTRunner's switch_to_window() to retry until the target is found in my draft as a temporary workaround.
• Chromium-specific testdriver-vendor.js implementation (not implemented).
  Chromium's run_web_tests.py doesn't work with the current PR yet.
  I tested my BFCache draft tests using testing/scripts/run_wpt_tests.py that uses official wpt runner and the tools/wptrunner/wptrunner/testdriver-extra.js etc. implementation in this PR.

Differences that can handled by pure JS code changes (mainly around navigations):

• Races between injected script evaluation and page load (e.g. inline script evaluation, window load events, pageshow events, etc.).
  • In the TestDriver version, injected scripts can be evaluated even before inline script evaluation.
• Promises returned by execute_script should not be resolved after navigation.
  • The COEP framework can wait for a Promise created before navigation and then resolved after navigation/back-navigation from BFCache.
• When to (and when not to) wait for script evaluation results
  • In the COEP framework, I intentionally did not send the injected script result back to the caller if the injected script initiates navigation, because sending the result back via server stash + Fetch API after triggering navigation might disable BFCache. Perhaps there are no such things in the TestDriver version?
  • I added a little more await for execute_script() results, because I encountered timeouts when I called execute_script() twice without awaiting for the first call (not investigated yet, but perhaps related to the single-threadedness of the WebDriver?).

Others:

• Chromium's testing/scripts/run_wpt_tests.py uses headless mode but BFCache is disabled in headless mode.
• Removing the headless flag enabled BFCache, but instead causes many more visibilitychange events, because switching tabs to find the target by WebDriver caused visibility changes.

[hiroshige-g]

Now I think I've figured out sufficient API specification to run BFCache tests at web-platform-tests/wpt#28950 (See /common/dispatcher/README.md).
This is basically equivalent with execute_script()-related parts of RFCs 88/89/91, plus additional clarifications around navigation (like #89 (comment)), minus testdriver integration.

If it looks good, then I'd like to land the Pure-JS-based RemoteContext.execute_script() and BFCache tests in

• [WPT] Move/merge COEP/COOP dispatcher framework to /common wpt#29684 and
• [WPT] Introduce RemoteContext.execute_script() and add basic BFCache tests + helpers wpt#28950

and continue discussions on test_driver integration, as I think we will have sufficient concensus about the API surface itself, except for whether it is integrated with test_driver, and BFCache tests based on this can be migrated between JS-based and test_driver-integrated versions of RemoteContext with minimum changes.

Does this sound good?

(underlying design doc for visibility, but most of the content is now included here and in web-platform-tests/wpt#28950)

[jgraham]

I don't have any objections to that. Note that #90 (comment) has a new proposal (which I'll write up properly this week) to provide cross-context execute script using only js. I think it's basically compatible with what you're doing, so it should be easy to update your code to that model in the future if desired.

[hiroshige-g]

Yeah it looks mostly compatible (I added some comments there but they don't affect much).
I'll aim to land the PRs relatively soon while I'll continue to follow the updates to the PRs.

[hiroshige-g]

cc/ @ArthurSonzogni.

My proposal is at #89 (comment), i.e. to have consensus around RFCs 88/89/91 plus BFCache-specific clarifications except for test_driver integration and RFC 90, which is summarized in web-platform-tests/wpt#28950.

I'd like to land this part soon and after that continue discussion around test_driver integration and RFC 90 (including jgraham@'s upcoming RFC/impl updates at #90 (comment)) separately, because BFCache tests don't directly depend on these two (BFCache tests are using send()/receive() indirectly, but they are encapsulated under RemoteContext.execute_script()).

[jgraham]

I'm withdrawning this in favour of #98 which has a non-WebDriver based approach to the same problem.