renderer/gl: add internal gl formats and reduce internal driver format conversions (#12879)

* format: add internal formats for drm formats

cross referenced with weston and added internal formats and types for a
lot of missing ones. also added a isFormatYUV helper.

* framebuffer: ensure we use right internalformat

ensure we use the right internal format to avoid internal driver
blitting, also since we only attach the GL_STENCIL_ATTACHMENT we might
just aswell only use the GL_STENCIL_INDEX8 to not confuse drivers that
we want a depth aswell.

* texture: use external on yuv or non linear mods

using external makes us use the gpu's internal detiler.
and this is makes intel a lot happier then having to format convert it
to a linear format internally.

* shaders: add external support to CM frag

add external support to CM frag, and correct ext.frag typo.

* formats: remove duplicates and fix a typo in cm.frag

remove duplicate formats and a typo in cm.frag

* formats: add swizzle logic to all formats

add swizzle logic from weston for all formats and use it in shm texture
paths.

* format: more format changes

use monitor drm format instead of forcing something different.

* shader: remove external from cm.frag

drivers want this resolved at compiletime cant use both
samplerExternalOES and sampler2d and then runtime branch it.

* screencopy: swizzle textures in screencopy

swizzle textures in screencopy, to get the right colors when copying.

* screencopy: restore old behaviour

try restore old behaviour before the gles3 format changes.
glReadPixels had the wrong format, so i went to far trying to mitigate
it. should be like before now.

1 files changed, 21 insertions, 6 deletions

diff --git a/src/protocols/Screencopy.cpp b/src/protocols/Screencopy.cpp
index c02b759c..d66c5342 100644
--- a/src/protocols/Screencopy.cpp
+++ b/src/protocols/Screencopy.cpp
@@ -192,8 +192,7 @@ void CScreencopyFrame::share() {
 }
 
 void CScreencopyFrame::renderMon() {
-    auto       TEXTURE = makeShared<CTexture>(m_monitor->m_output->state->state().buffer);
-
+    auto       TEXTURE    = makeShared<CTexture>(m_monitor->m_output->state->state().buffer);
     CRegion    fakeDamage = {0, 0, INT16_MAX, INT16_MAX};
 
     const bool IS_CM_AWARE = PROTO::colorManagement && PROTO::colorManagement->isClientCMAware(m_client->client());
@@ -385,8 +384,6 @@ bool CScreencopyFrame::copyShm() {
         return false;
     }
 
-    auto glFormat = PFORMAT->flipRB ? GL_BGRA_EXT : GL_RGBA;
-
     g_pHyprOpenGL->m_renderData.blockScreenShader = true;
     g_pHyprRenderer->endRender();
 
@@ -396,8 +393,26 @@ bool CScreencopyFrame::copyShm() {
 
     glPixelStorei(GL_PACK_ALIGNMENT, 1);
 
-    const auto drmFmt     = NFormatUtils::getPixelFormatFromDRM(shm.format);
-    uint32_t   packStride = NFormatUtils::minStride(drmFmt, m_box.w);
+    uint32_t packStride = NFormatUtils::minStride(PFORMAT, m_box.w);
+    int      glFormat   = PFORMAT->glFormat;
+
+    if (glFormat == GL_RGBA)
+        glFormat = GL_BGRA_EXT;
+
+    if (glFormat != GL_BGRA_EXT && glFormat != GL_RGB) {
+        if (PFORMAT->swizzle.has_value()) {
+            std::array<GLint, 4> RGBA = SWIZZLE_RGBA;
+            std::array<GLint, 4> BGRA = SWIZZLE_BGRA;
+            if (PFORMAT->swizzle == RGBA)
+                glFormat = GL_RGBA;
+            else if (PFORMAT->swizzle == BGRA)
+                glFormat = GL_BGRA_EXT;
+            else {
+                LOGM(Log::ERR, "Copied frame via shm might be broken or color flipped");
+                glFormat = GL_RGBA;
+            }
+        }
+    }
 
     // This could be optimized by using a pixel buffer object to make this async,
     // but really clients should just use a dma buffer anyways.