hyprtester: minor refactoring/restructure (#14154)

* hyprtester: Minor misc fixes

- Use `std::filesystem::path` instead of `std::string` in some places;
- Use `std::ranges::sort`, as suggested by clang-tidy;
- Reset colors at the end of every log line;
- Give more specific error messages in main.

* hyprtester: Fix a bug in `hyprlandAlive()`

Successful standard library calls are permitted to (and do) modify the
value of `errno` arbitrarily. Checking the value of `errno` only makes
sense when the library function returns an error. So, the correct
condition of a successful signal delivery is that the call to `kill`
returns 0.

Btw, given the possible other error values of `kill(2)`, should we even
consider `errno` at all?

* hyprtester: Restructure main

Split main's `main()` into multiple functions, separate responsibilities
more cleanly.

Add default values for cmd-line options to the help message.

* hyprtester: Remove redundant path canonicalization

* hyprtester: Run unified clean up code before every test

* tests: Use RAII for clients in client tests

Turn clients in client tests into classes. Constructors
initialize, destructors deinitialize.

Not only is it better design when entities (clients) are
modeled as classes, keeping all the methods connected,
this also ensures the tests perform their custom cleanup.

This will be especially important when we introduce early
termination on failure into the tests.

* hyprtester: Rewrite the testing framework!

- Implement a new test definition system used via the `TEST_CASE` and
  `SUBTEST` macros.

- Adds `ASSERT*` and reimplements `EXPECT*` macros. The former now
  terminate the test immediately. Also add some new macros, such as
  `FAIL_TEST`.

- Move existing tests to the new system. For trivial cases, break
  existing test functions down to multiple test cases.

- For non-trivial cases and a few other possible improvements, leave
  TODOs.

* tests: Use `EXPECT*` instead of `ASSERT*` in some places

`ASSERT*` macros terminate the test as soon as they fail. They are
appropriate to use when, if something fails, the rest of the test
does not make sense and more error messages from failing checks will
just pollute the output.

`EXPECT*` macros mark the test as failed but its execution continues.
It makes sense to use them when error messages from the following checks
in the test may be useful for someone who will be reading the log.

I changed some uses of `ASSERT*` in main tests to `EXPECT*`. But I was
not reading tests very carefully, so maybe I was wrong in some places.
My rule of thumb was:
- Checks that ensure that a kitty opens / a certain set/number of
  windows exists are fatal;
- A series of checks, especially on the same string returned by
  `Hyprland`, are non-fatal;
- Checks in tests with tightly coupled steps (i.e., do this, check,
  do this with the result of the previous step, check, etc) are fatal.

1 files changed, 41 insertions, 82 deletions

diff --git a/hyprtester/src/tests/main/groups.cpp b/hyprtester/src/tests/main/groups.cpp
index 71b5cf18..7e1c63b6 100644
--- a/hyprtester/src/tests/main/groups.cpp
+++ b/hyprtester/src/tests/main/groups.cpp
@@ -10,17 +10,14 @@
 #include <cerrno>
 #include "../shared.hpp"
 
-static int ret = 0;
-
 using namespace Hyprutils::OS;
 using namespace Hyprutils::Memory;
 
 #define UP CUniquePointer
 #define SP CSharedPointer
 
-static bool test() {
-    NLog::log("{}Testing groups", Colors::GREEN);
-
+// TODO: decompose this into multiple test cases
+TEST_CASE(groups) {
     // test on workspace "window"
     NLog::log("{}Dispatching workspace `groups`", Colors::YELLOW);
     getFromSocket("/dispatch hl.dsp.focus({ workspace = 'name:groups' })");
@@ -28,8 +25,7 @@ static bool test() {
     NLog::log("{}Testing movewindoworgroup from group to group", Colors::YELLOW);
     auto kittyA = Tests::spawnKitty("kittyA");
     if (!kittyA) {
-        NLog::log("{}Error: kitty did not spawn", Colors::RED);
-        return false;
+        FAIL_TEST("Could not spawn kitty");
     }
 
     // check kitty properties. One kitty should take the entire screen, minus the gaps.
@@ -43,8 +39,7 @@ static bool test() {
 
     auto kittyB = Tests::spawnKitty("kittyB");
     if (!kittyB) {
-        NLog::log("{}Error: kitty did not spawn", Colors::RED);
-        return false;
+        FAIL_TEST("Could not spawn kitty");
     }
 
     OK(getFromSocket("/dispatch hl.dsp.focus({ window = 'class:kittyB' })"));
@@ -61,8 +56,7 @@ static bool test() {
 
     auto kittyC = Tests::spawnKitty("kittyC");
     if (!kittyC) {
-        NLog::log("{}Error: kitty did not spawn", Colors::RED);
-        return false;
+        FAIL_TEST("Could not spawn kitty");
     }
 
     NLog::log("{}Check kittyC dimensions", Colors::YELLOW);
@@ -87,12 +81,11 @@ static bool test() {
     NLog::log("{}Spawning kittyProcA", Colors::YELLOW);
     auto kittyProcA = Tests::spawnKitty();
     if (!kittyProcA) {
-        NLog::log("{}Error: kitty did not spawn", Colors::RED);
-        return false;
+        FAIL_TEST("Could not spawn kitty");
     }
 
     NLog::log("{}Expecting 1 window", Colors::YELLOW);
-    EXPECT(Tests::windowCount(), 1);
+    ASSERT(Tests::windowCount(), 1);
 
     // check kitty properties. One kitty should take the entire screen, minus the gaps.
     NLog::log("{}Check kitty dimensions", Colors::YELLOW);
@@ -127,8 +120,7 @@ static bool test() {
     NLog::log("{}Spawn kitty again", Colors::YELLOW);
     kittyProcA = Tests::spawnKitty();
     if (!kittyProcA) {
-        NLog::log("{}Error: kitty did not spawn", Colors::RED);
-        return false;
+        FAIL_TEST("Could not spawn kitty");
     }
 
     NLog::log("{}Group kitty", Colors::YELLOW);
@@ -145,12 +137,11 @@ static bool test() {
     NLog::log("{}Spawn kittyProcB", Colors::YELLOW);
     auto kittyProcB = Tests::spawnKitty();
     if (!kittyProcB) {
-        NLog::log("{}Error: kitty did not spawn", Colors::RED);
-        return false;
+        FAIL_TEST("Could not spawn kitty");
     }
 
     NLog::log("{}Expecting 2 windows", Colors::YELLOW);
-    EXPECT(Tests::windowCount(), 2);
+    ASSERT(Tests::windowCount(), 2);
 
     size_t lastActiveKittyIdx = 0;
 
@@ -158,10 +149,7 @@ static bool test() {
     try {
         auto str           = getFromSocket("/activewindow");
         lastActiveKittyIdx = std::stoull(str.substr(7, str.find(" -> ") - 7), nullptr, 16);
-    } catch (...) {
-        NLog::log("{}Fail at getting prop", Colors::RED);
-        ret = 1;
-    }
+    } catch (...) { FAIL_TEST("Could not extract the active window id"); }
 
     // test cycling through
 
@@ -170,21 +158,15 @@ static bool test() {
 
     try {
         auto str = getFromSocket("/activewindow");
-        EXPECT(lastActiveKittyIdx != std::stoull(str.substr(7, str.find(" -> ") - 7), nullptr, 16), true);
-    } catch (...) {
-        NLog::log("{}Fail at getting prop", Colors::RED);
-        ret = 1;
-    }
+        ASSERT(lastActiveKittyIdx != std::stoull(str.substr(7, str.find(" -> ") - 7), nullptr, 16), true);
+    } catch (...) { FAIL_TEST("Could not extract the active window id"); }
 
     getFromSocket("/dispatch hl.dsp.group.next()");
 
     try {
         auto str = getFromSocket("/activewindow");
-        EXPECT(lastActiveKittyIdx, std::stoull(str.substr(7, str.find(" -> ") - 7), nullptr, 16));
-    } catch (...) {
-        NLog::log("{}Fail at getting prop", Colors::RED);
-        ret = 1;
-    }
+        ASSERT(lastActiveKittyIdx, std::stoull(str.substr(7, str.find(" -> ") - 7), nullptr, 16));
+    } catch (...) { FAIL_TEST("Could not extract the active window id"); }
 
     // test movegroupwindow: focus should follow the moved window
     NLog::log("{}Test movegroupwindow focus follows window", Colors::YELLOW);
@@ -195,10 +177,7 @@ static bool test() {
         str                  = getFromSocket("/activewindow");
         auto activeAfterMove = std::stoull(str.substr(7, str.find(" -> ") - 7), nullptr, 16);
         EXPECT(activeAfterMove, activeBeforeMove);
-    } catch (...) {
-        NLog::log("{}Fail at getting prop", Colors::RED);
-        ret = 1;
-    }
+    } catch (...) { FAIL_TEST("Could not extract the active window id"); }
 
     // and backwards
     NLog::log("{}Test movegroupwindow backwards", Colors::YELLOW);
@@ -209,10 +188,7 @@ static bool test() {
         str                  = getFromSocket("/activewindow");
         auto activeAfterMove = std::stoull(str.substr(7, str.find(" -> ") - 7), nullptr, 16);
         EXPECT(activeAfterMove, activeBeforeMove);
-    } catch (...) {
-        NLog::log("{}Fail at getting prop", Colors::RED);
-        ret = 1;
-    }
+    } catch (...) { FAIL_TEST("Could not extract the active window id"); }
 
     NLog::log("{}Disable autogrouping", Colors::YELLOW);
     OK(getFromSocket("/eval hl.config({ group = { auto_group = false } })"));
@@ -220,12 +196,11 @@ static bool test() {
     NLog::log("{}Spawn kittyProcC", Colors::YELLOW);
     auto kittyProcC = Tests::spawnKitty();
     if (!kittyProcC) {
-        NLog::log("{}Error: kitty did not spawn", Colors::RED);
-        return false;
+        FAIL_TEST("Could not spawn kitty");
     }
 
     NLog::log("{}Expecting 3 windows 2", Colors::YELLOW);
-    EXPECT(Tests::windowCount(), 3);
+    ASSERT(Tests::windowCount(), 3);
     {
         auto str = getFromSocket("/clients");
         EXPECT_COUNT_STRING(str, "at: 22,22", 2);
@@ -239,12 +214,11 @@ static bool test() {
     NLog::log("{}Spawn kittyProcD", Colors::YELLOW);
     auto kittyProcD = Tests::spawnKitty();
     if (!kittyProcD) {
-        NLog::log("{}Error: kitty did not spawn", Colors::RED);
-        return false;
+        FAIL_TEST("Could not spawn kitty");
     }
 
     NLog::log("{}Expecting 4 windows", Colors::YELLOW);
-    EXPECT(Tests::windowCount(), 4);
+    ASSERT(Tests::windowCount(), 4);
 
     OK(getFromSocket("/dispatch hl.dsp.group.active({ index = 3 })"));
 
@@ -258,7 +232,7 @@ static bool test() {
     Tests::killAllWindows();
 
     NLog::log("{}Expecting 0 windows", Colors::YELLOW);
-    EXPECT(Tests::windowCount(), 0);
+    ASSERT(Tests::windowCount(), 0);
 
     // test movewindoworgroup: direction should be respected when extracting from group
     NLog::log("{}Test movewindoworgroup respects direction out of group", Colors::YELLOW);
@@ -266,8 +240,7 @@ static bool test() {
     {
         auto kittyE = Tests::spawnKitty();
         if (!kittyE) {
-            NLog::log("{}Error: kitty did not spawn", Colors::RED);
-            return false;
+            FAIL_TEST("Could not spawn kitty");
         }
 
         // group kitty, and new windows should be auto-grouped
@@ -275,10 +248,9 @@ static bool test() {
 
         auto kittyF = Tests::spawnKitty();
         if (!kittyF) {
-            NLog::log("{}Error: kitty did not spawn", Colors::RED);
-            return false;
+            FAIL_TEST("Could not spawn kitty");
         }
-        EXPECT(Tests::windowCount(), 2);
+        ASSERT(Tests::windowCount(), 2);
 
         // both windows should be grouped at the same position
         {
@@ -310,7 +282,7 @@ static bool test() {
         }
 
         Tests::killAllWindows();
-        EXPECT(Tests::windowCount(), 0);
+        ASSERT(Tests::windowCount(), 0);
     }
 
     // test that we deny a floated window getting auto-grouped into a tiled group.
@@ -320,8 +292,7 @@ static bool test() {
     OK(getFromSocket("/eval hl.window_rule({ name = 'kitty-tiled', tile = true })"));
     auto kittyProcE = Tests::spawnKitty("kitty_tiled");
     if (!kittyProcE) {
-        NLog::log("{}Error: kitty did not spawn", Colors::RED);
-        return false;
+        FAIL_TEST("Could not spawn kitty");
     }
     OK(getFromSocket("/dispatch hl.dsp.group.toggle()"));
 
@@ -329,18 +300,16 @@ static bool test() {
     OK(getFromSocket("/eval hl.window_rule({ name = 'kitty-floated', float = true })"));
     auto kittyProcF = Tests::spawnKitty("kitty_floated");
     if (!kittyProcF) {
-        NLog::log("{}Error: kitty did not spawn", Colors::RED);
-        return false;
+        FAIL_TEST("Could not spawn kitty");
     }
 
-    EXPECT(Tests::windowCount(), 2);
+    ASSERT(Tests::windowCount(), 2);
 
     {
         auto clients  = getFromSocket("/clients");
         auto classPos = clients.find("class: kitty_floated");
         if (classPos == std::string::npos) {
-            NLog::log("{}Could not find kitty_floated in clients output", Colors::RED);
-            ret = 1;
+            FAIL_TEST("Could not find kitty_floated in clients output");
         } else {
             auto entryStart  = clients.rfind("Window ", classPos);
             auto entryEnd    = clients.find("\n\n", classPos);
@@ -351,27 +320,25 @@ static bool test() {
     }
 
     Tests::killAllWindows();
-    EXPECT(Tests::windowCount(), 0);
+    ASSERT(Tests::windowCount(), 0);
 
     // Tests for grouping/merging logic
     NLog::log("{}Testing locked groups w/ invade", Colors::GREEN);
 
     Tests::killAllWindows();
-    EXPECT(Tests::windowCount(), 0);
+    ASSERT(Tests::windowCount(), 0);
 
     // Test normal, unlocked groups
     {
         auto winA = Tests::spawnKitty("unlocked");
         if (!winA) {
-            NLog::log("{}Error: unlocked kitty did not spawn", Colors::RED);
-            return false;
+            FAIL_TEST("Could not spawn unlocked kitty");
         }
         OK(getFromSocket("/dispatch hl.dsp.group.toggle()"));
 
         auto winB = Tests::spawnKitty("top");
         if (!winB) {
-            NLog::log("{}Error: top kitty did not spawn", Colors::RED);
-            return false;
+            FAIL_TEST("Could not spawn top kitty");
         }
 
         // Verify it DID merge into a group
@@ -382,14 +349,13 @@ static bool test() {
     }
 
     Tests::killAllWindows();
-    EXPECT(Tests::windowCount(), 0);
+    ASSERT(Tests::windowCount(), 0);
 
     // Test locked groups
     {
         auto lockedWin = Tests::spawnKitty("locked");
         if (!lockedWin) {
-            NLog::log("{}Error: locked kitty did not spawn", Colors::RED);
-            return false;
+            FAIL_TEST("Could not spawn locked kitty");
         }
         OK(getFromSocket("/dispatch hl.dsp.group.toggle()"));
         OK(getFromSocket(std::format("/dispatch hl.dsp.focus({{ window = 'pid:{}' }})", lockedWin->pid())));
@@ -397,8 +363,7 @@ static bool test() {
 
         auto winB = Tests::spawnKitty("top");
         if (!winB) {
-            NLog::log("{}Error: top kitty did not spawn", Colors::RED);
-            return false;
+            FAIL_TEST("Could not spawn top kitty");
         }
 
         // Verify it did NOT merge into the locked group
@@ -409,7 +374,7 @@ static bool test() {
     }
 
     Tests::killAllWindows();
-    EXPECT(Tests::windowCount(), 0);
+    ASSERT(Tests::windowCount(), 0);
 
     // Test locked groups WITH invade rule
     {
@@ -418,14 +383,12 @@ static bool test() {
 
         auto lockedWin = Tests::spawnKitty("locked");
         if (!lockedWin) {
-            NLog::log("{}Error: locked kitty did not spawn", Colors::RED);
-            return false;
+            FAIL_TEST("Could not spawn locked kitty");
         }
 
         auto invadingWin = Tests::spawnKitty("invade");
         if (!invadingWin) {
-            NLog::log("{}Error: invading kitty did not spawn", Colors::RED);
-            return false;
+            FAIL_TEST("Could not spawn invading kitty");
         }
 
         // Verify it DID merge into the locked group
@@ -434,9 +397,5 @@ static bool test() {
     }
 
     Tests::killAllWindows();
-    EXPECT(Tests::windowCount(), 0);
-
-    return !ret;
+    ASSERT(Tests::windowCount(), 0);
 }
-
-REGISTER_TEST_FN(test)