Merge 646bd698db into df626de5ed

It now implements these commands.

.github/workflows/test-stable.yml

@@ -228,7 +228,7 @@ jobs:
       uses: actions/checkout@v3
       with:
         path: ngircd
-        ref: 0714466af88d71d6c395629cd7fb624b099507d4
+        ref: 3e3f6cbeceefd9357b53b27c2386bb39306ab353
         repository: ngircd/ngircd
     - name: Build ngircd
       run: |
@@ -1106,7 +1106,7 @@ jobs:
       uses: actions/checkout@v3
       with:
         path: sable
-        ref: fe337a036c3ab5f8548e2578b65568e628f4c32f
+        ref: b9deaa930c49f2939d9a584bedbfc3236da0d707
         repository: Libera-Chat/sable
     - name: Install rust toolchain
       uses: actions-rs/toolchain@v1

Makefile

@@ -93,7 +93,7 @@ SABLE_SELECTORS := \
 	and not arbitrary_client_tags \
 	and not react_tag \
 	and not private_chathistory \
-	and not whowas and not list and not lusers and not userhost and not time and not info \
+	and not list and not lusers and not time and not info \
 	$(EXTRA_SELECTORS)
 
 SOLANUM_SELECTORS := \

README.md

@@ -3,16 +3,30 @@
 This project aims at testing interoperability of software using the
 IRC protocol, by running them against common test suites.
 
+It is also used while editing [the "Modern" specification](https://modern.ircdocs.horse/)
+to check behavior of a large selection of servers at once.
+
 ## The big picture
 
 This project contains:
 
-* IRC protocol test cases
-* small wrappers around existing software to run tests on them
+* IRC protocol test cases, primarily checking conformance to
+  [the "Modern" specification](https://modern.ircdocs.horse/) and
+  [IRCv3 extensions](https://ircv3.net/irc/), but also
+  [RFC 1459](https://datatracker.ietf.org/doc/html/rfc1459) and
+  [RFC 2812](https://datatracker.ietf.org/doc/html/rfc2812).
+  Most of them are for servers but also some for clients.
+  Only the client-server protocol is tested; server-server protocols are out of scope.
+* Small wrappers around existing software to run tests on them.
+  So far this is restricted to headless software (servers, service packages,
+  and clients bots).
 
 Wrappers run software in temporary directories, so running `irctest` should
 have no side effect.
 
+Test results for the latest version of each supported software, and respective logs,
+are [published daily](https://dashboard.irctest.limnoria.net/).
+
 ## Prerequisites
 
 Install irctest and dependencies:
@@ -20,7 +34,7 @@ Install irctest and dependencies:
 ```
 sudo apt install faketime  # Optional, but greatly speeds up irctest/server_tests/list.py
 cd ~
-git clone https://github.com/ProgVal/irctest.git
+git clone https://github.com/progval/irctest.git
 cd irctest
 pip3 install --user -r requirements.txt
 ```
@@ -40,18 +54,23 @@ You can usually invoke it with `python3 -m pytest` command; which can often
 be called by the `pytest` or `pytest-3` commands (if not, alias them if you
 are planning to use them often).
 
+After installing `pytest-xdist`, you can also pass `pytest` the `-n 10` option
+to run `10` tests in parallel.
+
 The rest of this README assumes `pytest` works.
 
 ## Test selection
 
 A major feature of pytest that irctest heavily relies on is test selection.
 Using the `-k` option, you can select and deselect tests based on their names
-and/or markers (listed in `pytest.ini`).
 For example, you can run `LUSERS`-related tests with `-k lusers`.
-Or only tests based on RFC1459 with `-k rfc1459`.
+
+Using the `-m` option, you can select and deselect and them based on their markers
+(listed in `pytest.ini`).
+For example, you can run only tests based on RFC1459 with `-m rfc1459`.
 
 By default, all tests run; even niche ones. So you probably always want to
-use these options: `-k 'not Ergo and not deprecated and not strict`.
+use these options: `-m 'not Ergo and not deprecated and not strict`.
 This excludes:
 
 * `Ergo`-specific tests (included as Ergo uses irctest as its official
@@ -63,6 +82,10 @@ This excludes:
 
 ## Running tests
 
+This list is non-exhaustive, see `workflows.yml` for software not listed here.
+If software you want to test is not listed their either, please open an issue
+or pull request to add support for it.
+
 ### Servers
 
 #### Ergo:
@@ -89,20 +112,6 @@ make install
 pytest --controller irctest.controllers.solanum -k 'not Ergo and not deprecated and not strict'
 ```
 
-#### Charybdis:
-
-```
-cd /tmp/
-git clone https://github.com/atheme/charybdis.git
-cd charybdis
-./autogen.sh
-./configure --prefix=$HOME/.local/
-make -j 4
-make install
-cd ~/irctest
-pytest --controller irctest.controllers.charybdis -k 'not Ergo and not deprecated and not strict'
-```
-
 #### InspIRCd:
 
 ```
@@ -123,14 +132,6 @@ cd ~/irctest
 pytest --controller irctest.controllers.inspircd -k 'not Ergo and not deprecated and not strict'
 ```
 
-#### Mammon:
-
-```
-pip3 install --user git+https://github.com/mammon-ircd/mammon.git
-cd ~/irctest
-pytest --controller irctest.controllers.mammon -k 'not Ergo and not deprecated and not strict'
-```
-
 #### UnrealIRCd:
 
 ```
@@ -147,8 +148,8 @@ pytest --controller irctest.controllers.unreal -k 'not Ergo and not deprecated a
 
 ### Servers with services
 
-Besides Ergo (that has built-in services), most server controllers can optionally run
-service packages.
+Besides Ergo (that has built-in services) and Sable (that ships its own services),
+most server controllers can optionally run service packages.
 
 #### Atheme:
 

irctest/controllers/sable.py

@@ -116,20 +116,7 @@ NETWORK_CONFIG_CONFIG = """
     ],
 
     "alias_users": [
-        {
-            "nick": "ChanServ",
-            "user": "ChanServ",
-            "host": "services.",
-            "realname": "Channel services compatibility layer",
-            "command_alias": "CS"
-        },
-        {
-            "nick": "NickServ",
-            "user": "NickServ",
-            "host": "services.",
-            "realname": "Account services compatibility layer",
-            "command_alias": "NS"
-        }
+        %(services_alias_users)s
     ],
 
     "default_roles": {
@@ -160,6 +147,23 @@ NETWORK_CONFIG_CONFIG = """
 }
 """
 
+SERVICES_ALIAS_USERS = """
+        {
+            "nick": "ChanServ",
+            "user": "ChanServ",
+            "host": "services.",
+            "realname": "Channel services compatibility layer",
+            "command_alias": "CS"
+        },
+        {
+            "nick": "NickServ",
+            "user": "NickServ",
+            "host": "services.",
+            "realname": "Account services compatibility layer",
+            "command_alias": "NS"
+        }
+"""
+
 SERVER_CONFIG = """
 {
     "server_id": 1,
@@ -374,6 +378,7 @@ class SableController(BaseServerController, DirectoryBasedController):
             .strip(),
             services_management_hostname=services_management_hostname,
             services_management_port=services_management_port,
+            services_alias_users=SERVICES_ALIAS_USERS if run_services else "",
         )
 
         with self.open_file("configs/network.conf") as fd:

irctest/server_tests/names.py

@@ -68,6 +68,39 @@ class NamesTestCase(cases.BaseServerTestCase):
         """
         self._testNames(symbol=True, allow_trailing_space=False)
 
+    @cases.mark_specifications("RFC2812", "Modern")
+    def testNames2812Secret(self):
+        """The symbol sent for a secret channel is `@` instead of `=`:
+        https://datatracker.ietf.org/doc/html/rfc2812#section-3.2.5
+        https://modern.ircdocs.horse/#rplnamreply-353
+        """
+        self.connectClient("nick1")
+        self.sendLine(1, "JOIN #chan")
+        # enable secret channel mode
+        self.sendLine(1, "MODE #chan +s")
+        self.getMessages(1)
+        self.sendLine(1, "NAMES #chan")
+        messages = self.getMessages(1)
+        self.assertMessageMatch(
+            messages[0],
+            command=RPL_NAMREPLY,
+            params=["nick1", "@", "#chan", StrRe("@nick1 ?")],
+        )
+        self.assertMessageMatch(
+            messages[1],
+            command=RPL_ENDOFNAMES,
+            params=["nick1", "#chan", ANYSTR],
+        )
+
+        self.connectClient("nick2")
+        self.sendLine(2, "JOIN #chan")
+        namreplies = [msg for msg in self.getMessages(2) if msg.command == RPL_NAMREPLY]
+        self.assertNotEqual(len(namreplies), 0)
+        for msg in namreplies:
+            self.assertMessageMatch(
+                msg, command=RPL_NAMREPLY, params=["nick2", "@", "#chan", ANYSTR]
+            )
+
     def _testNamesMultipleChannels(self, symbol):
         self.connectClient("nick1")
 

irctest/server_tests/who.py

@@ -87,7 +87,7 @@ class BaseWhoTestCase:
 class WhoTestCase(BaseWhoTestCase, cases.BaseServerTestCase):
     @cases.mark_specifications("Modern")
     def testWhoStar(self):
-        if self.controller.software_name in ("Bahamut", "Sable"):
+        if self.controller.software_name in ("Bahamut",):
             raise runner.OptionalExtensionNotSupported("WHO mask")
 
         self._init()
@@ -118,7 +118,7 @@ class WhoTestCase(BaseWhoTestCase, cases.BaseServerTestCase):
     )
     @cases.mark_specifications("Modern")
     def testWhoNick(self, mask):
-        if "*" in mask and self.controller.software_name in ("Bahamut", "Sable"):
+        if "*" in mask and self.controller.software_name in ("Bahamut",):
             raise runner.OptionalExtensionNotSupported("WHO mask")
 
         self._init()
@@ -148,7 +148,7 @@ class WhoTestCase(BaseWhoTestCase, cases.BaseServerTestCase):
         ids=["username", "realname-mask", "hostname"],
     )
     def testWhoUsernameRealName(self, mask):
-        if "*" in mask and self.controller.software_name in ("Bahamut", "Sable"):
+        if "*" in mask and self.controller.software_name in ("Bahamut",):
             raise runner.OptionalExtensionNotSupported("WHO mask")
 
         self._init()
@@ -201,7 +201,7 @@ class WhoTestCase(BaseWhoTestCase, cases.BaseServerTestCase):
     )
     @cases.mark_specifications("Modern")
     def testWhoNickAway(self, mask):
-        if "*" in mask and self.controller.software_name in ("Bahamut", "Sable"):
+        if "*" in mask and self.controller.software_name in ("Bahamut",):
             raise runner.OptionalExtensionNotSupported("WHO mask")
 
         self._init()
@@ -235,7 +235,7 @@ class WhoTestCase(BaseWhoTestCase, cases.BaseServerTestCase):
     )
     @cases.mark_specifications("Modern")
     def testWhoNickOper(self, mask):
-        if "*" in mask and self.controller.software_name in ("Bahamut", "Sable"):
+        if "*" in mask and self.controller.software_name in ("Bahamut",):
             raise runner.OptionalExtensionNotSupported("WHO mask")
 
         self._init()
@@ -274,7 +274,7 @@ class WhoTestCase(BaseWhoTestCase, cases.BaseServerTestCase):
     )
     @cases.mark_specifications("Modern")
     def testWhoNickAwayAndOper(self, mask):
-        if "*" in mask and self.controller.software_name in ("Bahamut", "Sable"):
+        if "*" in mask and self.controller.software_name in ("Bahamut",):
             raise runner.OptionalExtensionNotSupported("WHO mask")
 
         self._init()
@@ -308,7 +308,7 @@ class WhoTestCase(BaseWhoTestCase, cases.BaseServerTestCase):
     @pytest.mark.parametrize("mask", ["#chan", "#CHAN"], ids=["exact", "casefolded"])
     @cases.mark_specifications("Modern")
     def testWhoChan(self, mask):
-        if "*" in mask and self.controller.software_name in ("Bahamut", "Sable"):
+        if "*" in mask and self.controller.software_name in ("Bahamut",):
             raise runner.OptionalExtensionNotSupported("WHO mask")
 
         self._init()
@@ -632,7 +632,7 @@ class WhoServicesTestCase(BaseWhoTestCase, cases.BaseServerTestCase):
 class WhoInvisibleTestCase(cases.BaseServerTestCase):
     @cases.mark_specifications("Modern")
     def testWhoInvisible(self):
-        if self.controller.software_name in ("Bahamut", "Sable"):
+        if self.controller.software_name in ("Bahamut",):
             raise runner.OptionalExtensionNotSupported("WHO mask")
 
         self.connectClient("evan", name="evan")

irctest/server_tests/whowas.py

@@ -154,6 +154,9 @@ class WhowasTestCase(cases.BaseServerTestCase):
         except ConnectionClosed:
             pass
 
+        if self.controller.software_name == "Sable":
+            time.sleep(1)  # may take a little while to record the historical user
+
         self.sendLine(1, whowas_command)
 
         messages = self.getMessages(1)

workflows.yml

@@ -230,7 +230,7 @@ software:
         name: ngircd
         repository: ngircd/ngircd
         refs:
-            stable: 0714466af88d71d6c395629cd7fb624b099507d4 # two years ahead of rel-26.1
+            stable: 3e3f6cbeceefd9357b53b27c2386bb39306ab353 # three years ahead of rel-26.1
             release: null
             devel: master
             devel_release: null
@@ -249,7 +249,7 @@ software:
         name: Sable
         repository: Libera-Chat/sable
         refs:
-            stable: fe337a036c3ab5f8548e2578b65568e628f4c32f
+            stable: b9deaa930c49f2939d9a584bedbfc3236da0d707
             release: null
             devel: master
             devel_release: null