Agent CA rotation: in-band certificate renewal + fix gem install on Ruby 2.7

.github/workflows/ci.yml

@@ -34,7 +34,27 @@
       - name: Run tests
         run: bundle exec rspec
 
+  # `bundle exec rspec` runs against the checkout and so never resolves the
+  # gemspec's dependencies the way a customer does. That gap let timers 4.4.0
+  # (required_ruby_version >= 3.1) satisfy a '~> 4.3' constraint and break
+  # `gem install deploy-agent` outright on every Ruby below 3.1, with a green
+  # CI throughout. This job closes it: build the gem and install it for real on
+  # the oldest Ruby the gemspec claims to support.
+  gem-install:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: '2.7'
+      - name: Build and install the gem
+        run: |
+          gem build deploy-agent.gemspec
+          gem install --no-document ./deploy-agent-*.gem
+      - name: Run the installed executable
+        run: deploy-agent version
+
   release-please:
     runs-on: ubuntu-latest
     if: github.ref == 'refs/heads/master'
     outputs:
@@ -46,7 +66,7 @@
 
   release:
     runs-on: ubuntu-latest
-    needs: [lint, test, release-please]
+    needs: [lint, test, gem-install, release-please]
     if: ${{ needs.release-please.outputs.release_created }}
     steps:
       - uses: actions/checkout@v4

README.md

@@ -1,6 +1,6 @@
 # Deploy Agent
 
-> **DEPRECATED:** This gem is deprecated and will not receive further updates.
+> **DEPRECATED:** This gem is deprecated and only receives essential fixes.
 > Please migrate to the new [network-agent](https://github.com/deployhq/network-agent) instead,
 > which has fewer dependencies and is easier to install.
 >
@@ -31,6 +31,18 @@ The agent connects **outbound** to DeployHQ, so no inbound firewall rules are ne
 gem install deploy-agent
 ```
 
+## Upgrading
+
+```bash
+gem install deploy-agent
+deploy-agent restart
+```
+
+Use `gem install`, not `gem update`. On Ruby versions below 3.1 a released gem
+could not have its dependencies resolved, and `gem update deploy-agent` responds
+to that by reporting `Gems already up-to-date` and installing nothing. `gem
+install` resolves from scratch and picks up the fix.
+
 ## Quick Start
 
 ### 1. Configure the agent
@@ -114,6 +126,26 @@ To allow the agent to connect to additional servers, edit `~/.deploy/agent.acces
 
 Lines starting with `#` are comments. Each entry can be an individual IP address or a CIDR network range.
 
+## Certificate renewal
+
+DeployHQ is rotating the certificate authority behind the agent connection. Each
+time the agent connects it asks whether a replacement client certificate is
+waiting for it, and installs one if DeployHQ offers it.
+
+This is automatic and needs no action from you. The agent keeps its identity —
+same name, same configured servers, nothing to re-claim — and simply reconnects
+once using the new certificate.
+
+A replacement is only written after it has been checked against the agent's
+existing private key and the certificate authorities the agent ships with. If
+any check fails, the agent logs a warning, keeps the certificate it already has
+and carries on. A successful renewal is logged as `Certificate renewed` at the
+default log level, in `~/.deploy/agent.log` when the agent runs in the background.
+
+What matters on your side is staying up to date: an agent still presenting a
+certificate issued by the old authority after **17 March 2027** will not be able
+to connect. See [Upgrading](#upgrading).
+
 ## Troubleshooting
 
 **Agent won't start — "not configured"**

ca.crt

@@ -31,3 +31,33 @@ l4CPcGbB0L8yyIhGwiEfrZpjx6hOelX1daG8QPTvSSYpB6ODtQeb3zpDf8vU8M7T
 oAwG8/0g1Owh/a970vIKu4TBa4D2IiCfA3KPWlsIUSoeu9uBTKmUQ0Raa0AhZWPv
 JI4XgcL63KznYzLm0BOxvTYMxDfn7fs=
 -----END CERTIFICATE-----
+-----BEGIN CERTIFICATE-----
+MIIFGzCCAwOgAwIBAgIBATANBgkqhkiG9w0BAQsFADAvMRowGAYDVQQDDBFEZXBs
+b3lIUSBBZ2VudCBDQTERMA8GA1UECgwIRGVwbG95SFEwHhcNMjYwOTE1MDczODI4
+WhcNMzYwOTE1MDczODI4WjAvMRowGAYDVQQDDBFEZXBsb3lIUSBBZ2VudCBDQTER
+MA8GA1UECgwIRGVwbG95SFEwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoIC
+AQC4bMAQgGZC7ZUO9Wq2+rWodOXL4IZ6Ae+Dfce+SlTsQUx5HrYrmMFTE9NJTxcE
+dGYjAVVwpdvgT8flFKRiQa7j1xVOLr73ra7Uro0ZGHw1TOBFFAhXLZ25oXT8SsB7
+1aaRZksZAPGR1ZkWRvKqUkRge9LERtcmZDOSeKyf3i3PQp3AaODA4tRwsQnyKWoA
+6h9Ruk9jnHqNRapiEjjdkubKKRjgU28IO0Abom5dRsDvvFYVquaQmud745kdaxDr
+fQA9Kyz8lzNSglmTHACjzxtX1AGRpn5yLO7PqaV0M/QqnKZhaKW+tzP3pWdsDZ8v
+RBGDhyVYeyHYSPAEOgsRdTqQYpnXkR1FLLS/FCFQ/UgyWFStFVv3l9mzq/fQLr/O
+sdPzG9KNNTz4OqMrGOeRMcBXsKQJjIRS6TSzggPagcxT2iPRHCGQknko8/O3IyM1
+9YIsLm7kxw2oEIzhHx8Wv3s+SafJr4cuTu1n9RfyP3LmU/C6GbfrH/jDvVuLvFdH
+9AY55RNCfCgCKWjqDeYrzGxIcuoZ57H6DJlGouw0MiSg4hUUqNnM1EQG410bJouA
+itprB7FIGcZ9uHQvpEgs8NP/DTRsmUP+51Vdp6SLbFSNrt5v/APDrd3ya8R3f7g0
+XVLFl/jxNM4Ki+L6ZGwRz4SYThew0OZXsezbnYiGEro1JwIDAQABo0IwQDAdBgNV
+HQ4EFgQUgnCdcTQCEbmADry95bq7UAoX71AwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
+HQ8BAf8EBAMCAQYwDQYJKoZIhvcNAQELBQADggIBAHaN63Zqs4W5FTTqQPYnkMH0
+/hMzRVVcQ8tZng6rGy4HdONixQ2XOwFOCMbl8uqRB/ihyLbwiLIVgsqG3wMEIiqV
+lxNDYerUtdBolam/lqA371d18lEqKFiFfncCRrd+AzTMcYnYZZK026smx1JCxDCP
+t1rB/T9Lv0g2aRyWTKsKBeI5GFOIEIHScP1k/R7549kNEwerFKX5USWszDfnouiV
+hSz9UjlY1vQQ//qYgimevJxlGudt2b+73+BkT5THdNKBxeyn02/DbhaH/1zafE/7
+lxAKmN94HkAOM2TD27ZPRDIK4jmKK/Hx03X0kYD+KL/9Mr/4PIjlObRFF8BUawp1
+Kpn/Caw4gNz76IujGXGvJp9FxsZUscRRRdikIZ3AtR0/6Wyf8fFaAd89DoBGSzLM
+mRynJbnK5ROlVzvxX43ow7sBLtoh1+/RCsWcCVgOC2EMCW2n4b5M0USv7FBrhPie
+F50NkMqsTtEoxw8kcKvLd4FbPvWuIxPsXzxlnFaHTbVgTevgL6O8IVeWLqdqwFGK
+owB7z1tEQTN6sV4+fDDtaVQFCalEV9QPAolVLYdBUApc866heDHmzW85tkq2zoDG
+2cj80hCcI42DPvrb9nEcIcZiFZDCcIlDjRf6kDmqBW4oXr5V1yAxrC+i0UA2+LWj
+NVgD1tDYCLJlNY3GA6j8
+-----END CERTIFICATE-----

deploy-agent.gemspec

@@ -20,10 +20,14 @@ Gem::Specification.new do |s|
 
   s.add_dependency 'nio4r', '~> 2.7'
   s.add_dependency 'rb-readline', '~> 0.5'
-  s.add_dependency 'timers', '~> 4.3'
+  # timers 4.4.0 raised required_ruby_version to >= 3.1. '~> 4.3' admits it, and the
+  # RubyGems shipped with Ruby 2.7 cannot back off to 4.3.5 on its own, so a plain
+  # `gem install deploy-agent` fails outright on every Ruby this gem still supports
+  # below 3.1. Keep the upper bound until required_ruby_version moves past 3.1.
+  s.add_dependency 'timers', '>= 4.3', '< 4.4'
 
   s.post_install_message = <<~MSG
-    WARNING: deploy-agent is deprecated and will not receive further updates.
+    WARNING: deploy-agent is deprecated and only receives essential fixes.
     Please migrate to the new agent: https://github.com/deployhq/network-agent
   MSG
 end

lib/deploy_agent.rb

@@ -1,4 +1,5 @@
 require 'deploy_agent/version'
+require 'deploy_agent/certificate_renewal'
 require 'deploy_agent/configuration_generator'
 require 'deploy_agent/server_connection'
 require 'deploy_agent/destination_connection'

lib/deploy_agent/certificate_renewal.rb

@@ -0,0 +1,136 @@
+# frozen_string_literal: true
+
+require 'openssl'
+
+module DeployAgent
+  # Validates and atomically installs a replacement client certificate offered by
+  # the Deploy server over the tunnel (see ServerConnection::COMMAND_RENEW_RESPONSE).
+  #
+  # A renewal is a re-signature of the certificate we already hold: the server
+  # re-signs our stored public key under a different CA, keeping the subject and
+  # the serial. Our private key is never involved, so a renewed certificate must
+  # still pair with the agent.key already on disk.
+  #
+  # Every check runs before anything touches the filesystem, and the replacement
+  # is swapped in with rename(2). A bad agent.crt is not a recoverable state: the
+  # agent would fail the TLS handshake on every reconnect and Agent#run gives up
+  # and exits the process after four consecutive SSL errors.
+  class CertificateRenewal
+
+    class InvalidCertificate < StandardError; end
+
+    def initialize(certificate_path: CERTIFICATE_PATH, key_path: KEY_PATH, ca_path: CA_PATH)
+      @certificate_path = certificate_path
+      @key_path = key_path
+      @ca_path = ca_path
+    end
+
+    # Validate a PEM-encoded replacement certificate and install it.
+    #
+    # Returns the installed OpenSSL::X509::Certificate, or nil when the server
+    # offered the certificate we are already using (in which case nothing is
+    # written and there is no reason to reconnect).
+    #
+    # Raises InvalidCertificate - having written nothing at all - if any check
+    # fails. The caller keeps its working certificate.
+    def install(pem)
+      new_certificate = parse(pem)
+      validate!(new_certificate)
+      return nil if new_certificate.to_der == current_certificate.to_der
+
+      write(new_certificate)
+      new_certificate
+    end
+
+    private
+
+    def parse(pem)
+      raise InvalidCertificate, 'renewal response contained no certificate' if pem.nil? || pem.empty?
+
+      OpenSSL::X509::Certificate.new(pem)
+    rescue OpenSSL::OpenSSLError => e
+      raise InvalidCertificate, "could not parse the offered certificate: #{e.message}"
+    end
+
+    def validate!(new_certificate)
+      # Renewal re-signs our public key, it never re-keys us, so the replacement
+      # has to pair with the private key we already hold.
+      unless new_certificate.check_private_key(private_key)
+        raise InvalidCertificate, 'offered certificate does not match the agent private key'
+      end
+
+      # The backend identifies this agent by the certificate serial, and the
+      # subject carries the agent name. Neither may change across a renewal.
+      unless new_certificate.subject == current_certificate.subject
+        raise InvalidCertificate,
+              "offered certificate has a different subject (#{current_certificate.subject} -> #{new_certificate.subject})"
+      end
+
+      unless new_certificate.serial == current_certificate.serial
+        raise InvalidCertificate,
+              "offered certificate has a different serial (#{current_certificate.serial} -> #{new_certificate.serial})"
+      end
+
+      # And it has to chain to a CA we ship *as a TLS client certificate*, or we
+      # would be trading a working certificate for one the server refuses on the
+      # next handshake - and every handshake after it.
+      store = certificate_store
+      return if store.verify(new_certificate)
+
+      raise InvalidCertificate, "offered certificate is not a usable client certificate (#{store.error_string})"
+    end
+
+    # Write to a private temporary file in the same directory, flush it all the
+    # way to disk, then rename over agent.crt so a reader never sees a partial
+    # certificate and a crash mid-write cannot destroy the working one.
+    def write(certificate)
+      temp_path = temporary_path
+      begin
+        File.open(temp_path, File::WRONLY | File::CREAT | File::EXCL, 0o600) do |file|
+          file.write(certificate.to_pem)
+          file.flush
+          file.fsync
+        end
+        File.rename(temp_path, @certificate_path)
+      rescue StandardError
+        File.unlink(temp_path) if File.file?(temp_path)
+        raise
+      end
+    end
+
+    def temporary_path
+      directory = File.dirname(@certificate_path)
+      basename = File.basename(@certificate_path)
+      File.join(directory, ".#{basename}.#{Process.pid}.#{rand(0xffffffff).to_s(16)}")
+    end
+
+    def current_certificate
+      @current_certificate ||= OpenSSL::X509::Certificate.new(File.read(@certificate_path))
+    rescue SystemCallError, OpenSSL::OpenSSLError => e
+      raise InvalidCertificate, "could not read the current certificate: #{e.message}"
+    end
+
+    def private_key
+      @private_key ||= OpenSSL::PKey::RSA.new(File.read(@key_path))
+    rescue SystemCallError, OpenSSL::OpenSSLError => e
+      raise InvalidCertificate, "could not read the agent private key: #{e.message}"
+    end
+
+    def certificate_store
+      @certificate_store ||= OpenSSL::X509::Store.new.tap do |store|
+        store.add_file(@ca_path)
+        # The agent presents this certificate for TLS client authentication, so
+        # verify it for that purpose rather than the store's default, which
+        # accepts anything that merely chains. Real agent certificates carry no
+        # extensions at all and this purpose accepts them; what it rejects is a
+        # certificate whose keyUsage or extendedKeyUsage rules client auth out
+        # (a serverAuth-only certificate, say), which would otherwise install
+        # cleanly and then be refused by the server on every reconnect.
+        store.purpose = OpenSSL::X509::PURPOSE_SSL_CLIENT
+      end
+    rescue SystemCallError, OpenSSL::OpenSSLError => e
+      raise InvalidCertificate, "could not read the CA bundle: #{e.message}"
+    end
+
+  end
+end

lib/deploy_agent/cli.rb

@@ -6,7 +6,7 @@ class CLI
 
     DEPRECATION_NOTICE = <<~MSG
       \e[33m╔══════════════════════════════════════════════════════════════════╗
-      ║  DEPRECATED: deploy-agent will not receive further updates.      ║
+      ║  DEPRECATED: deploy-agent only receives essential fixes.         ║
       ║  Please migrate to the new agent:                                ║
       ║                                                                  ║
       ║    https://github.com/deployhq/network-agent                     ║

lib/deploy_agent/server_connection.rb

@@ -9,6 +9,17 @@ class ServerDisconnected < StandardError;end
     attr_reader :destination_connections, :agent
     attr_writer :nio_monitor
 
+    # Tunnel commands. 1-7 are the original proxy protocol. 8 and 9 were added
+    # for in-band certificate renewal and are implemented identically in the Go
+    # agent (network-agent) and in the backend.
+    COMMAND_RENEW_REQUEST  = 8 # agent -> server, payload "ruby/<version>"
+    COMMAND_RENEW_RESPONSE = 9 # server -> agent, payload [status:1][body]
+
+    # COMMAND_RENEW_RESPONSE statuses
+    RENEW_STATUS_RENEWED = 0 # body is the replacement certificate, PEM encoded
+    RENEW_STATUS_CURRENT = 1 # no body, the certificate we hold is current
+    RENEW_STATUS_ERROR   = 2 # body is a UTF-8 message
+
     # Create a secure TLS connection to the Deploy server
     def initialize(agent, server_host, nio_selector, check_certificate=true)
       @agent = agent
@@ -40,6 +51,12 @@ def initialize(agent, server_host, nio_selector, check_certificate=true)
       @nio_monitor = @nio_selector.register(@tcp_socket, :r)
       @nio_monitor.value = self
 
+      # Ask the server whether a replacement certificate is waiting for us. The
+      # server decides and answers with COMMAND_RENEW_RESPONSE; it never sends one
+      # unsolicited. This has to come after the monitor exists because send_packet
+      # arms it for writing.
+      request_certificate_renewal
+
       @agent.logger.info "Successfully connected to server"
     rescue => e
       @agent.logger.info "Something went wrong connecting to server."
@@ -111,6 +128,9 @@ def rx_data
           # This is a shutdown request. Disconnect and don't re-attempt connection.
           @agent.logger.warn "Server requested reconnect. Closing connection."
           close
+        when COMMAND_RENEW_RESPONSE
+          # The server has answered our renewal request.
+          handle_renewal_response(packet[1..-1])
         end
       end
     rescue EOFError, Errno::ECONNRESET, Errno::ETIMEDOUT, Errno::ENETRESET
@@ -183,6 +203,48 @@ def close
       raise ServerDisconnected
     end
 
+    # Ask the server to re-issue our client certificate, telling it which agent
+    # implementation and version is asking. Renewal is best effort: it must never
+    # stop us connecting, so a failure here is logged and otherwise ignored.
+    def request_certificate_renewal
+      @agent.logger.debug "Requesting certificate renewal"
+      send_packet([COMMAND_RENEW_REQUEST, "ruby/#{DeployAgent::VERSION}"].pack('Ca*'))
+    rescue => e
+      @agent.logger.warn "Could not request certificate renewal: #{e.message}"
+    end
+
+    # Process a COMMAND_RENEW_RESPONSE. Renewal must never take the agent down, so
+    # anything unexpected is logged and the existing certificate is kept.
+    def handle_renewal_response(body)
+      body = body.to_s
+      status = body.bytes[0]
+      payload = body[1..-1].to_s
+
+      case status
+      when RENEW_STATUS_RENEWED
+        certificate = CertificateRenewal.new.install(payload)
+        if certificate
+          @agent.logger.info "Certificate renewed (issuer=#{certificate.issuer})"
+          # Reconnect so the new certificate is the one we present. close raises
+          # ServerDisconnected, which Agent#run catches and retries, and the new
+          # connection re-reads agent.crt from disk.
+          close
+        else
+          @agent.logger.debug "Server offered the certificate we already hold"
+        end
+      when RENEW_STATUS_CURRENT
+        @agent.logger.debug "Certificate is up to date"
+      when RENEW_STATUS_ERROR
+        @agent.logger.warn "Server could not renew our certificate: #{payload}"
+      else
+        @agent.logger.warn "Unknown certificate renewal status: #{status.inspect}"
+      end
+    rescue ServerDisconnected
+      raise
+    rescue => e
+      @agent.logger.warn "Certificate renewal failed: #{e.message}"
+    end
+
     # Queue a packet of data to be sent to the Deploy server
     def send_packet(data)
       @tx_buffer << [data.bytesize+2, data].pack('na*')