Suggest User Agents treat a captured tab as visible

[jan-ivar]

From @youennf in w3c/mediacapture-surface-control#42:

I had a quick try and AIUI, Chrome is making sure that a tab thinks it is visible (which is sort of true) when captured.
This would also be consistent with unsuspending/ not-throttling the captured tab.
And this also helps ensuring that the surface control API is not leaking that a tab is being captured.

This might be good input for the screen share main spec.

Chrome's behavior here seems desirable, so we might want to capture it in the spec, as either mandate or best practice.

[jan-ivar]

@youennf did you check if the captured tab's visibility was tied to the capturing tab's visibility?

To determine whether it should be normative or not might depend on whether this would be considered JS observable or not, which seems debatable.

While thinking through a WPT test for this — instrument self-capture and then window.open() another tab and to see if focus still is lost — is when it occurred to me it would make sense for the visibility to be inherited, which would make the WPT not work. So maybe this is not observable, and we should just describe it as best practice?

[jan-ivar]

As to whether this should include broader guidance against leaking that a tab is being captured, discuss.

[youennf]

@youennf did you check if the captured tab's visibility was tied to the capturing tab's visibility?

I didn't, given it can be sent to remote peers, my guess it that it does not depend on the capturing tab's visibility.
It would be nice to get input from Chrome's implementors.

To determine whether it should be normative or not might depend on whether this would be considered JS observable

This is JS observable via document.visibilitystate and its related change event.

broader guidance against leaking that a tab is being captured, discuss.

If there is no guidance, I think it is worth adding a section describing the potential issues.

[alvestrand]

Slides for Jan 21 meeting suggest to expand the scope of this issue to include Window. I think that's unwise, given that capturing a Window is capturing the application at OS level, and we don't have special considerations for browsers when doing that; it's not even clear that we can have. With tab capture, it is obvious how to connect the signalling path.

[youennf]

For window, there is indeed tighter integration with the OS.
I wonder whether guidance could help in some cases though, say if browser is capturing one of its own window.

[eladalon1983]

The suggestion in Slides for Jan 21 meeting makes sense to me. I think it'd make sense to stick to tabs for now; capturing a window that happens to be a browser window is interesting, but there is no immediate use case that requires a decision here, allowing us more time to think.

[jan-ivar]

This seems like a purely technical distinction. Tabs and windows seem interchangeable to most users. All browsers today allow users to drag loose a tab into its own window, or collapse windows back to adjacent tabs.

I think we should capture this intent in the spec. Happy to discuss SHOULD vs MUST.

FWIW Firefox is aware of capturing its own window, even showing a warning in its permission prompt (on platforms without integrated pickers).

[alvestrand]

When we capture from the "window" list in Chrome, we are doing a window capture.
When we capture from the "tab" list in Chrome, we're capturing a tab.
At least in Chrome, there doesn't seem to be a confusion.

(And Chrome doesn't produce frames from a window capture if the window's minimized)

[jan-ivar]

I'm happy we seem to have agreement on a MUST specifically for tab capture.

There was pushback in the meeting on window capture, so I wanted to explain how I got there, to see if we can continue discussion on that part.

I wonder whether guidance could help in some cases though, say if browser is capturing one of its own window.

If I right-click on this fiddle link and open it in a new browser window, it'll dump document.visibilityState every 2 seconds .

I found two ways to make it emit "hidden" in all browsers (tested on mac and windows):

1. Open a new tab in that browser window, or
2. drag another larger window on top of that browser window to 100% occlude it

document.visibilityState = "visible"
document.visibilityState = "visible"
document.visibilityState = "hidden"
document.visibilityState = "hidden"

So the user agent is aware the page is hidden in both cases. If it's also aware of its browser window being captured by itself (like Firefox is), then is there a reason Firefox couldn't emit "visible" instead?

Is the OS throttling the window process in the second case? If so, that might be a convincing argument to not emit "visible", as it would backfire and potentially leak capture.

[eladalon1983]

Tabs and windows seem interchangeable to most users.

1. Do you have user studies supporting this, or is this based on personal opinion?
2. There are different technical issues associated with either. One is platform-independent, the other is not.
3. I am not aware of a guarantee of a 1:1 relation between browser windows and browser tabs. What guarantees that a browser window does not have 0 or 10 associated "visible" tabs?

[youennf]

As I said during the meeting, I hope we can write a guideline that would apply to all display surfaces.
Maybe something like below:

• If the capture source of a hidden surface is not producing frames, there is no need to prevent throttling.
• Conversely, generation of frames should prevent throttling.
• If preventing throttling is done, the surface should be considered as visible (specific to UA surfaces probably).

I would tend to prefer should here since this is mostly about quality of experience, and tradeoffs may vary from one UA to the other, or from one OS to the other.

[eladalon1983]

I would tend to prefer should here since this is mostly about quality of experience, and tradeoffs may vary from one UA to the other, or from one OS to the other.

Level of throttling might differ too. Keeping it guideline-level would leave UAs valuable discretion here.

[jan-ivar]

I would tend to prefer should here since this is mostly about quality of experience ...

The OP says it's also about ensuring that the surface control API is not leaking that a tab is being captured.

For instance, the following output from https://jsfiddle.net/jib1/519jp0ht/show would reveal the tab is likely being captured:

document.visibilityState = "visible" window.devicePixelRatio = 2
document.visibilityState = "hidden" window.devicePixelRatio = 2
document.visibilityState = "hidden" window.devicePixelRatio = 2
document.visibilityState = "hidden" window.devicePixelRatio = 2.200000047683716
document.visibilityState = "hidden" window.devicePixelRatio = 2.200000047683716

[youennf]

the surface control API is not leaking that a tab is being captured.

It is worth mentioning that specific point in the privacy section.
I am unsure how much we can enforce this in the spec though.

[jan-ivar]

I am unsure how much we can enforce this in the spec though.

I think we can enforce it in w3c/mediacapture-surface-control#42 (comment).

• If the capture source of a hidden surface is not producing frames, there is no need to prevent throttling.
• Conversely, generation of frames should prevent throttling.
• If preventing throttling is done, the surface should be considered as visible (specific to UA surfaces probably).

This sounds reasonable, except throttling is more of a UA optimization. Can we express it without that?

Based on this note in html:

...maybe we should express it relative to the document's visibility state instead?

Happy to review a PR.

[dontcallmedom-bot]

This issue had an associated resolution in WebRTC January 2025 meeting – 21 January 2025 (Suggest User Agents treat a captured tab as visible):

RESOLUTION: focus on tab capture and make it a requirement (MUST)