renderer/gl: add internal gl formats and reduce internal driver format conversions (#12879)

* format: add internal formats for drm formats

cross referenced with weston and added internal formats and types for a
lot of missing ones. also added a isFormatYUV helper.

* framebuffer: ensure we use right internalformat

ensure we use the right internal format to avoid internal driver
blitting, also since we only attach the GL_STENCIL_ATTACHMENT we might
just aswell only use the GL_STENCIL_INDEX8 to not confuse drivers that
we want a depth aswell.

* texture: use external on yuv or non linear mods

using external makes us use the gpu's internal detiler.
and this is makes intel a lot happier then having to format convert it
to a linear format internally.

* shaders: add external support to CM frag

add external support to CM frag, and correct ext.frag typo.

* formats: remove duplicates and fix a typo in cm.frag

remove duplicate formats and a typo in cm.frag

* formats: add swizzle logic to all formats

add swizzle logic from weston for all formats and use it in shm texture
paths.

* format: more format changes

use monitor drm format instead of forcing something different.

* shader: remove external from cm.frag

drivers want this resolved at compiletime cant use both
samplerExternalOES and sampler2d and then runtime branch it.

* screencopy: swizzle textures in screencopy

swizzle textures in screencopy, to get the right colors when copying.

* screencopy: restore old behaviour

try restore old behaviour before the gles3 format changes.
glReadPixels had the wrong format, so i went to far trying to mitigate
it. should be like before now.

1 files changed, 20 insertions, 2 deletions

diff --git a/src/protocols/ToplevelExport.cpp b/src/protocols/ToplevelExport.cpp
index 9a97f934..7549425c 100644
--- a/src/protocols/ToplevelExport.cpp
+++ b/src/protocols/ToplevelExport.cpp
@@ -285,8 +285,6 @@ bool CToplevelExportFrame::copyShm(const Time::steady_tp& now) {
     glBindFramebuffer(GL_READ_FRAMEBUFFER, outFB.getFBID());
     glPixelStorei(GL_PACK_ALIGNMENT, 1);
 
-    auto glFormat = PFORMAT->flipRB ? GL_BGRA_EXT : GL_RGBA;
-
     auto origin = Vector2D(0, 0);
     switch (PMONITOR->m_transform) {
         case WL_OUTPUT_TRANSFORM_FLIPPED_180:
@@ -308,6 +306,26 @@ bool CToplevelExportFrame::copyShm(const Time::steady_tp& now) {
         default: break;
     }
 
+    int glFormat = PFORMAT->glFormat;
+
+    if (glFormat == GL_RGBA)
+        glFormat = GL_BGRA_EXT;
+
+    if (glFormat != GL_BGRA_EXT && glFormat != GL_RGB) {
+        if (PFORMAT->swizzle.has_value()) {
+            std::array<GLint, 4> RGBA = SWIZZLE_RGBA;
+            std::array<GLint, 4> BGRA = SWIZZLE_BGRA;
+            if (PFORMAT->swizzle == RGBA)
+                glFormat = GL_RGBA;
+            else if (PFORMAT->swizzle == BGRA)
+                glFormat = GL_BGRA_EXT;
+            else {
+                LOGM(Log::ERR, "Copied frame via shm might be broken or color flipped");
+                glFormat = GL_RGBA;
+            }
+        }
+    }
+
     glReadPixels(origin.x, origin.y, m_box.width, m_box.height, glFormat, PFORMAT->glType, pixelData);
 
     if (overlayCursor) {