clickhouse-keeper on notebook1 is unreachable by replicas

[aagbsn]

due to how notebook1 is configured, the clickhouse service is only listening on lo, which means that only outgoing connections work and clients who try to contact notebook1 are unable to do so. There doesn't appear to be a convenient way to set a different listening address for the clickhouse-keeper service separately from the replica components; the proposal is to replace notebook1's role as a clickhouse-keeper with clickhouse2.

Separately I noticed that when trying to update the clickhouse configuration on notebook1 it complained that the keeper ID was inconsistent with the notebook.ooni.org hostname - that change was made in devops end of 2025, so I'm not sure for how long this configuration change was rejected.

It does appear that there was originally a clickhouse-keeper with ID 2 pointed at clickhouse2.prod.ooni.io; which is the same name as the new instance. This node is no longer part of the raft consensus, but I am not sure if there will be any issues with retaining the same hostname / ID or changing the ID.

The steps to roll out this change:

(TODO: think about the state of raft consensus as updates are applied, given that notebook1's availability isn't certain)
...

1. update clickhouse2's configuration, adding it as a clickhouse-keeper.
2. update notebook1's config, adding clickhouse2 as a clickhouse-keeper. check the logs and see if it complains
3. update clickhouse3 with the same change. at this point, notebook1, clickhouse3, and clickhouse2 should agree and there should be a consensus between these hosts, but clickhouse1 is not yet part of the consensus.
4. update clickhouse1 with the same changes. there should now be full quorum with 4 nodes.
5. at some point of stability, notebook1 is removed from the set of keeper nodes, and there is a quorum with 3 keepers.

[aagbsn]

because notebook1 may not be reachable; it might make sense to apply the changes to this node last; then wait for a quorum of 3 before dropping notebook1.