#2026 - http.agent.name is dropped from the robots.txt match set when http.robots.agents is set

[rzo1]

Closes #2026.

RobotRulesParser.setConf() never adds http.agent.name to agentNames when http.robots.agents is non-empty:

• if the advertised name is listed first, index++ skips it and the loop appends only agents[1..], so the name is dropped entirely;
• if it is not listed first, the code logs "Agent we advertise (X) not listed first" and appends the configured list, still without adding the name.

agentNames is only written in the empty-list branch and in that loop, which contradicts the in-code comment: "If both are present, our agent-string should be the first one we advertise to robots-parsing."

Impact. With http.agent.name: mybot and http.robots.agents: mybot,mybot-uk, the match set handed to crawler-commons is [mybot-uk]. A User-agent: mybot group in a robots.txt does not match and the parser falls back to *. Rules addressed specifically to the crawler are ignored, which is a compliance issue rather than a cosmetic one.

Fix. Add agentNames.add(agentName) before the append loop. agentNames is a LinkedHashSet, so the existing index++ stays as-is and no duplicate is introduced.

Two commits: the failing reproducer first, then the fix - happy to squash before merge.

For all changes

• Is there a issue associated with this PR? Is it referenced in the commit message?
• Does your PR title start with #XXXX where XXXX is the issue number you are trying to resolve?
• Has your PR been rebased against the latest commit within the target branch (typically main)?
• Is your initial contribution a single, squashed commit? - two commits, red reproducer then fix
• Is the code properly formatted with mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false? - not run yet

For code changes

• Have you ensured that the full suite of tests is executed via mvn clean verify? - only the protocol robots tests were run locally
• Have you written or updated unit tests to verify your changes? - new RobotRulesParserAgentNamesTest (4 tests, 3 of which fail without the fix)
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0? - no new dependencies
• If applicable, have you updated the LICENSE file, including the main LICENSE file? - n/a
• If applicable, have you updated the NOTICE file, including the main NOTICE file? - n/a